Ehcache replication exceptions in JIRA DataCenter

Hi,

I've setup an JIRA Datacenter (based on 2 JIRA Software 7.2.0) like shown here https://confluence.atlassian.com/adminjiraserver071/installing-jira-data-center-802592197.html

I'm seeing a lot of exceptions in the log regarding ehcache replication.

Both nodes are active and responding. Load balancer (Apache2) can access and properly proxy to both instances. All configs are installation default - I've changed only $JIRA_INSTALL/conf/server.xml (added jvmRoute and HTTPS handling).

Ports open on both machines: 8080 (JIRA), 8000 (debug), 40001 (ehcache).

How can I fix this?

Exception below:

2017-02-23 10:14:44,397 myWorkHostChecker:thread-1 ERROR      [n.s.ehcache.distribution.RMISynchronousCacheReplicator] Exception on replication of putNotification. RemoteException occurred in server thread; nested exception is:
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled). Continuing...
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:355)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
        at net.sf.ehcache.distribution.RMICachePeer_Stub.put(Unknown Source)
        at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:149)
        at com.atlassian.cache.ehcache.replication.rmi.RMISynchronousCacheReplicator.replicateViaCopy(RMISynchronousCacheReplicator.java:60)
        at com.atlassian.cache.ehcache.replication.rmi.RMISynchronousCacheReplicator.notifyElementPut(RMISynchronousCacheReplicator.java:48)
        at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementPut(RegisteredEventListeners.java:192)
        at net.sf.ehcache.event.RegisteredEventListeners.notifyElementPut(RegisteredEventListeners.java:170)
        at net.sf.ehcache.Cache.notifyPutInternalListeners(Cache.java:1648)
        at net.sf.ehcache.Cache.putInternal(Cache.java:1618)
        at net.sf.ehcache.Cache.put(Cache.java:1543)
        at net.sf.ehcache.Cache.put(Cache.java:1508)
        at com.atlassian.cache.ehcache.DelegatingCache.put(DelegatingCache.java:82)
        at com.atlassian.mywork.client.service.HostIdCache.setHost(HostIdCache.java:47)
        at com.atlassian.mywork.client.service.HostServiceImpl.setRegisteredHost(HostServiceImpl.java:211)
        at com.atlassian.mywork.client.service.HostServiceImpl.updateHostAvailability(HostServiceImpl.java:200)
        at com.atlassian.mywork.client.service.HostServiceImpl.enable(HostServiceImpl.java:102)
        at com.atlassian.mywork.client.service.ServiceSelectorImpl$2.run(ServiceSelectorImpl.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
        java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        ... 3 more
Caused by: java.lang.ClassNotFoundException: com.atlassian.mywork.client.service.HostServiceImpl$HostType (no security manager: RMI class loader disabled)
        at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
        at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186)
        at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
        at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
        at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:219)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
        at java.io.ObjectInputStream.readEnum(ObjectInputStream.java:1733)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
        at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:503)
        at net.sf.ehcache.Element.readObject(Element.java:884)
        ... 2 filtered
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
        at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:326)
        at sun.rmi.server.UnicastServerRef.unmarshalParametersUnchecked(UnicastServerRef.java:592)
        at sun.rmi.server.UnicastServerRef.unmarshalParameters(UnicastServerRef.java:580)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:310)
        ... 12 more

2 answers

1 vote
Chris Fuller Atlassian Team Feb 23, 2017

That looks to me like the MyWork plugin is attempting to use its own class as a cache key.  This does not work, because cache replication occurs over RMI and, as the exception indicates, the RMI class loader restricts which classes it will load for security reasons (and plugin classes are not available to it).  This should be considered a bug in the MyWork plugin, which is owned internally within Atlassian by Confluence's development team.

Please report this to support.

0 vote

There is a bug record opened to address this matter which was fixed on JIRA Sever 7.2.6, please refer: https://jira.atlassian.com/browse/JRA-59979.

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published Thursday in Jira

Meet the AUG leaders of Northern Virginia

@Rachel Wright (Jira Genie), @Billy Poggi (AUG NOVA, DC), and @Dana Jansen (Confluence Queen) are just some of the folks that lead one of the world's most active Atlassian User Group (AUG)....

125 views 4 9
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you