java.rmi.ConnectException: Connection refused to host in Atlassian Jira Data Center

[#1]

I had to prepare Atlassian Jira Data Center environment for performance testing according to this guide. One of the steps is to restore Atlassian backup to the Jira environment. After I restored the backup my log was full of java.rmi.ConnectException: Connection refused to host and I could connect to Atlassian Jira only using the recovery_admin user.

Here is the complete error:

2020-03-24 06:34:09,388+0000 localq-reader-44 INFO      [c.a.j.c.distribution.localq.LocalQCacheOpReader] Checked exception: RecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.servicedesk.internal.feature.customer.portal.PortalInternalManagerImpl.allPortalsCache', action=REMOVE, key=com.atlassian.cache.impl.ReferenceKey@0, value=null, creationTimeInMillis=1584848666452} from cache replication queue: [queueId=queue_node4_2_9e22b2ee283109ab44b3ddeb56f9ed7a, queuePath=/opt/atlassian/jira/data-8.0.2/localq/queue_node4_2_9e22b2ee283109ab44b3ddeb56f9ed7a], failuresCount: 600. Will retry indefinitely.
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$RecoverableFailure: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:86)
	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
	... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 15 more
2020-03-24 06:34:09,389+0000 localq-reader-42 INFO      [c.a.j.c.distribution.localq.LocalQCacheOpReader] Checked exception: RecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.cache', action=REMOVE, key=CacheKey[entityName=jira.properties,entityId=1], value=null, creationTimeInMillis=1584848685070} from cache replication queue: [queueId=queue_node4_0_9e22b2ee283109ab44b3ddeb56f9ed7a, queuePath=/opt/atlassian/jira/data-8.0.2/localq/queue_node4_0_9e22b2ee283109ab44b3ddeb56f9ed7a], failuresCount: 600. Will retry indefinitely.
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$RecoverableFailure: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:86)
	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
	... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 15 more

When I tried to login the login form hanged for a couple of minutes and then threw the incorrect user or password error.

I investigated the problem and discovered that I had more nodes than it was supposed to be:

I deleted the wrong nodes from the clusternode table:

delete FROM "clusternode" where node_id in ('mycluster1',  'i-074313de8fe42329f', 'i-0ecc34529ac845f3f' , 'i-0621d3ecdb771021f', 'i-0c9dfca4d5f793eab')

After it the error disappeared and I could connect to Jira.

Hope it will help you to solve this error.

[Taranjeet Singh]

nice useful information @Alexey Matveev !