Ehcache replication exceptions in JIRA DataCenter

Mateusz Waś February 22, 2017

Hi,

I've setup an JIRA Datacenter (based on 2 JIRA Software 7.2.0) like shown here https://confluence.atlassian.com/adminjiraserver071/installing-jira-data-center-802592197.html

I'm seeing a lot of exceptions in the log regarding ehcache replication.

Both nodes are active and responding. Load balancer (Apache2) can access and properly proxy to both instances. All configs are installation default - I've changed only $JIRA_INSTALL/conf/server.xml (added jvmRoute and HTTPS handling).

Ports open on both machines: 8080 (JIRA), 8000 (debug), 40001 (ehcache).

How can I fix this?

Exception below:

2017-02-23 10:14:44,397 myWorkHostChecker:thread-1 ERROR      [n.s.ehcache.distribution.RMISynchronousCacheReplicator] Exception on replication of putNotification. RemoteException occurred in server thread; nested exception is:
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled). Continuing...
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:355)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
        at net.sf.ehcache.distribution.RMICachePeer_Stub.put(Unknown Source)
        at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:149)
        at com.atlassian.cache.ehcache.replication.rmi.RMISynchronousCacheReplicator.replicateViaCopy(RMISynchronousCacheReplicator.java:60)
        at com.atlassian.cache.ehcache.replication.rmi.RMISynchronousCacheReplicator.notifyElementPut(RMISynchronousCacheReplicator.java:48)
        at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementPut(RegisteredEventListeners.java:192)
        at net.sf.ehcache.event.RegisteredEventListeners.notifyElementPut(RegisteredEventListeners.java:170)
        at net.sf.ehcache.Cache.notifyPutInternalListeners(Cache.java:1648)
        at net.sf.ehcache.Cache.putInternal(Cache.java:1618)
        at net.sf.ehcache.Cache.put(Cache.java:1543)
        at net.sf.ehcache.Cache.put(Cache.java:1508)
        at com.atlassian.cache.ehcache.DelegatingCache.put(DelegatingCache.java:82)
        at com.atlassian.mywork.client.service.HostIdCache.setHost(HostIdCache.java:47)
        at com.atlassian.mywork.client.service.HostServiceImpl.setRegisteredHost(HostServiceImpl.java:211)
        at com.atlassian.mywork.client.service.HostServiceImpl.updateHostAvailability(HostServiceImpl.java:200)
        at com.atlassian.mywork.client.service.HostServiceImpl.enable(HostServiceImpl.java:102)
        at com.atlassian.mywork.client.service.ServiceSelectorImpl$2.run(ServiceSelectorImpl.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        ... 3 more
Caused by: java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
        at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186)
        at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
        at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
        at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:219)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
        at java.io.ObjectInputStream.readEnum(ObjectInputStream.java:1733)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:503)
        at net.sf.ehcache.Element.readObject(Element.java:884)
        ... 2 filtered
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
        at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:326)
        at sun.rmi.server.UnicastServerRef.unmarshalParametersUnchecked(UnicastServerRef.java:592)
        at sun.rmi.server.UnicastServerRef.unmarshalParameters(UnicastServerRef.java:580)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:310)
        ... 12 more

2 answers

2 votes
crf
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 23, 2017

That looks to me like the MyWork plugin is attempting to use its own class as a cache key.  This does not work, because cache replication occurs over RMI and, as the exception indicates, the RMI class loader restricts which classes it will load for security reasons (and plugin classes are not available to it).  This should be considered a bug in the MyWork plugin, which is owned internally within Atlassian by Confluence's development team.

Please report this to support.

0 votes
Luciano Fagundes
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 28, 2017

There is a bug record opened to address this matter which was fixed on JIRA Sever 7.2.6, please refer: https://jira.atlassian.com/browse/JRA-59979.

Suggest an answer

Log in or Sign up to answer