What Jira operations cause synchronous activity in RMISynchronousCacheReplicator?

dmitri_z5hhjq

We suffered a brief network partition in our JIRA Datacenter setup. Among other exceptions, we noticed the usage of RMISynchronousCacheReplicator in some stacktraces in http threads. When do user facing threads cause synchronous activity? Can this be configured to be asynchronous?

Stacktrace from our incident:

java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
        at net.sf.ehcache.distribution.RMICachePeer_Stub.remove(Unknown Source)
        at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicateRemovalNotification(RMISynchronousCacheReplicator.java:243)
        at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementRemoved(RMISynchronousCacheReplicator.java:229)
        at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementRemoved(RegisteredEventListeners.java:144)
        at net.sf.ehcache.event.RegisteredEventListeners.notifyElementRemoved(RegisteredEventListeners.java:124)
        at net.sf.ehcache.Cache.notifyRemoveInternalListeners(Cache.java:2322)
        at net.sf.ehcache.Cache.removeInternal(Cache.java:2305)
        at net.sf.ehcache.Cache.remove(Cache.java:2207)
        at net.sf.ehcache.Cache.remove(Cache.java:2125)
        at net.sf.ehcache.constructs.EhcacheDecoratorAdapter.remove(EhcacheDecoratorAdapter.java:154)
        at com.atlassian.cache.ehcache.LoadingCache.remove(LoadingCache.java:208)
        at com.atlassian.cache.ehcache.DelegatingCache.remove(DelegatingCache.java:135)
        at com.atlassian.jira.avatar.CachingTaggingAvatarStore.getByIdTagged(CachingTaggingAvatarStore.java:104)
        at com.atlassian.jira.avatar.AvatarManagerImpl.getByIdTagged(AvatarManagerImpl.java:117)
        at com.atlassian.jira.avatar.AvatarManagerImpl.lambda$transformToJIRAAvatar$480(AvatarManagerImpl.java:421)
        at com.atlassian.fugue.Option$Some.fold(Option.java:421)
        at com.atlassian.fugue.Option.flatMap(Option.java:287)
        at com.atlassian.jira.avatar.AvatarManagerImpl.transformToJIRAAvatar(AvatarManagerImpl.java:421)
        at com.atlassian.jira.avatar.AvatarManagerImpl.processAvatarData(AvatarManagerImpl.java:376)
        at com.atlassian.jira.avatar.AvatarManagerImpl.readAvatarData(AvatarManagerImpl.java:488)
        at com.atlassian.jira.web.servlet.AvatarToStream.sendAvatar(AvatarToStream.java:29)
        at com.atlassian.jira.web.servlet.AbstractAvatarServlet.defaultDoGet(AbstractAvatarServlet.java:82)
        at com.atlassian.jira.web.servlet.ViewUserAvatarServlet.defaultDoGet(ViewUserAvatarServlet.java:39)
        at com.atlassian.jira.web.servlet.AbstractAvatarServlet.doGet(AbstractAvatarServlet.java:39)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

crf

When do user facing threads cause synchronous activity?

Essentially, any write operation to a synchronous cache or the initial access to a replicated cache that has not yet bootstrapped values from another node.  At one point in time the Users and Groups caches behaved this way, but I think that was fixed in 7.0.  I have 7.1.0 checked out and can tell by the code that is definitely fixed there, so the only cache I know would do this is the TaskManager's.

Can this be configured to be asynchronous?

It definitely can in the code:

    https://docs.atlassian.com/atlassian-cache-api/2.11.3/atlassian-cache-api/apidocs/index.html

I could be wrong, but my understanding is that we use asynchronous replication by default and that only caches that explicitly opt in to being synchronous will replicate in that manner.  The only such cache I'm aware of offhand is the one for the TaskManager, which is one of the few value-replicating caches in JIRA (most replicate invalidation only).

You should be able to tell from a thread dump which cache was doing synchronous replication.  Without knowing which caches were involved, it isn't really possible to fairly evaluate what the potential consequences of changing them from synchronous to asynchronous might be.  It could be that the cache's author was just being paranoid, or it could be that asynchronous replication destabilized behaviour for their code, so they specified synchronous replication as a workaround.