Ehcache replication exceptions in JIRA DataCenter

[Mateusz Waś]

Hi,

I've setup an JIRA Datacenter (based on 2 JIRA Software 7.2.0) like shown here https://confluence.atlassian.com/adminjiraserver071/installing-jira-data-center-802592197.html

I'm seeing a lot of exceptions in the log regarding ehcache replication.

Both nodes are active and responding. Load balancer (Apache2) can access and properly proxy to both instances. All configs are installation default - I've changed only $JIRA_INSTALL/conf/server.xml (added jvmRoute and HTTPS handling).

Ports open on both machines: 8080 (JIRA), 8000 (debug), 40001 (ehcache).

How can I fix this?

Exception below:

2017-02-23 10:14:44,397 myWorkHostChecker:thread-1 ERROR      [n.s.ehcache.distribution.RMISynchronousCacheReplicator] Exception on replication of putNotification. RemoteException occurred in server thread; nested exception is:
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled). Continuing...
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:355)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
        at net.sf.ehcache.distribution.RMICachePeer_Stub.put(Unknown Source)
        at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:149)
        at com.atlassian.cache.ehcache.replication.rmi.RMISynchronousCacheReplicator.replicateViaCopy(RMISynchronousCacheReplicator.java:60)
        at com.atlassian.cache.ehcache.replication.rmi.RMISynchronousCacheReplicator.notifyElementPut(RMISynchronousCacheReplicator.java:48)
        at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementPut(RegisteredEventListeners.java:192)
        at net.sf.ehcache.event.RegisteredEventListeners.notifyElementPut(RegisteredEventListeners.java:170)
        at net.sf.ehcache.Cache.notifyPutInternalListeners(Cache.java:1648)
        at net.sf.ehcache.Cache.putInternal(Cache.java:1618)
        at net.sf.ehcache.Cache.put(Cache.java:1543)
        at net.sf.ehcache.Cache.put(Cache.java:1508)
        at com.atlassian.cache.ehcache.DelegatingCache.put(DelegatingCache.java:82)
        at com.atlassian.mywork.client.service.HostIdCache.setHost(HostIdCache.java:47)
        at com.atlassian.mywork.client.service.HostServiceImpl.setRegisteredHost(HostServiceImpl.java:211)
        at com.atlassian.mywork.client.service.HostServiceImpl.updateHostAvailability(HostServiceImpl.java:200)
        at com.atlassian.mywork.client.service.HostServiceImpl.enable(HostServiceImpl.java:102)
        at com.atlassian.mywork.client.service.ServiceSelectorImpl$2.run(ServiceSelectorImpl.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        ... 3 more
Caused by: java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
        at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186)
        at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
        at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
        at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:219)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
        at java.io.ObjectInputStream.readEnum(ObjectInputStream.java:1733)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:503)
        at net.sf.ehcache.Element.readObject(Element.java:884)
        ... 2 filtered
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
        at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:326)
        at sun.rmi.server.UnicastServerRef.unmarshalParametersUnchecked(UnicastServerRef.java:592)
        at sun.rmi.server.UnicastServerRef.unmarshalParameters(UnicastServerRef.java:580)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:310)
        ... 12 more

[crf]

That looks to me like the MyWork plugin is attempting to use its own class as a cache key.  This does not work, because cache replication occurs over RMI and, as the exception indicates, the RMI class loader restricts which classes it will load for security reasons (and plugin classes are not available to it).  This should be considered a bug in the MyWork plugin, which is owned internally within Atlassian by Confluence's development team.

Please report this to support.

[Luciano Fagundes]

There is a bug record opened to address this matter which was fixed on JIRA Sever 7.2.6, please refer: https://jira.atlassian.com/browse/JRA-59979.