java.rmi.ConnectException: Connection refused to host in Atlassian Jira Data Center

I had to prepare Atlassian Jira Data Center environment for performance testing according to this guide. One of the steps is to restore Atlassian backup to the Jira environment. After I restored the backup my log was full of java.rmi.ConnectException: Connection refused to host and I could connect to Atlassian Jira only using the recovery_admin user.

Here is the complete error:

2020-03-24 06:34:09,388+0000 localq-reader-44 INFO      [c.a.j.c.distribution.localq.LocalQCacheOpReader] Checked exception: RecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.servicedesk.internal.feature.customer.portal.PortalInternalManagerImpl.allPortalsCache', action=REMOVE, key=com.atlassian.cache.impl.ReferenceKey@0, value=null, creationTimeInMillis=1584848666452} from cache replication queue: [queueId=queue_node4_2_9e22b2ee283109ab44b3ddeb56f9ed7a, queuePath=/opt/atlassian/jira/data-8.0.2/localq/queue_node4_2_9e22b2ee283109ab44b3ddeb56f9ed7a], failuresCount: 600. Will retry indefinitely.
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$RecoverableFailure: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:86)
	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
	... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 15 more
2020-03-24 06:34:09,389+0000 localq-reader-42 INFO      [c.a.j.c.distribution.localq.LocalQCacheOpReader] Checked exception: RecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.cache', action=REMOVE, key=CacheKey[entityName=jira.properties,entityId=1], value=null, creationTimeInMillis=1584848685070} from cache replication queue: [queueId=queue_node4_0_9e22b2ee283109ab44b3ddeb56f9ed7a, queuePath=/opt/atlassian/jira/data-8.0.2/localq/queue_node4_0_9e22b2ee283109ab44b3ddeb56f9ed7a], failuresCount: 600. Will retry indefinitely.
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$RecoverableFailure: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:86)
	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
	... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 15 more

When I tried to login the login form hanged for a couple of minutes and then threw the incorrect user or password error.

I investigated the problem and discovered that I had more nodes than it was supposed to be:

Screenshot 2020-03-24 at 09.24.33.png

I deleted the wrong nodes from the clusternode table:

delete FROM "clusternode" where node_id in ('mycluster1',  'i-074313de8fe42329f', 'i-0ecc34529ac845f3f' , 'i-0621d3ecdb771021f', 'i-0c9dfca4d5f793eab')

After it the error disappeared and I could connect to Jira.

Hope it will help you to solve this error.

1 comment

Taranjeet Singh
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
March 25, 2020

nice useful information @Alexey Matveev !

Comment

Log in or Sign up to comment
TAGS
AUG Leaders

Atlassian Community Events