It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

java.rmi.ConnectException: Connection refused to host in Atlassian Jira Data Center

I had to prepare Atlassian Jira Data Center environment for performance testing according to this guide. One of the steps is to restore Atlassian backup to the Jira environment. After I restored the backup my log was full of java.rmi.ConnectException: Connection refused to host and I could connect to Atlassian Jira only using the recovery_admin user.

Here is the complete error:

2020-03-24 06:34:09,388+0000 localq-reader-44 INFO      [c.a.j.c.distribution.localq.LocalQCacheOpReader] Checked exception: RecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.servicedesk.internal.feature.customer.portal.PortalInternalManagerImpl.allPortalsCache', action=REMOVE, key=com.atlassian.cache.impl.ReferenceKey@0, value=null, creationTimeInMillis=1584848666452} from cache replication queue: [queueId=queue_node4_2_9e22b2ee283109ab44b3ddeb56f9ed7a, queuePath=/opt/atlassian/jira/data-8.0.2/localq/queue_node4_2_9e22b2ee283109ab44b3ddeb56f9ed7a], failuresCount: 600. Will retry indefinitely.
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$RecoverableFailure: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:86)
	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
	... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 15 more
2020-03-24 06:34:09,389+0000 localq-reader-42 INFO      [c.a.j.c.distribution.localq.LocalQCacheOpReader] Checked exception: RecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.cache', action=REMOVE, key=CacheKey[entityName=jira.properties,entityId=1], value=null, creationTimeInMillis=1584848685070} from cache replication queue: [queueId=queue_node4_0_9e22b2ee283109ab44b3ddeb56f9ed7a, queuePath=/opt/atlassian/jira/data-8.0.2/localq/queue_node4_0_9e22b2ee283109ab44b3ddeb56f9ed7a], failuresCount: 600. Will retry indefinitely.
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$RecoverableFailure: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:86)
	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.ConnectException: Connection refused to host: prod-1-jira-node1.internal; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
	at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
	... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
	at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 15 more

When I tried to login the login form hanged for a couple of minutes and then threw the incorrect user or password error.

I investigated the problem and discovered that I had more nodes than it was supposed to be:

Screenshot 2020-03-24 at 09.24.33.png

I deleted the wrong nodes from the clusternode table:

delete FROM "clusternode" where node_id in ('mycluster1',  'i-074313de8fe42329f', 'i-0ecc34529ac845f3f' , 'i-0621d3ecdb771021f', 'i-0c9dfca4d5f793eab')

After it the error disappeared and I could connect to Jira.

Hope it will help you to solve this error.

1 comment

nice useful information @Alexey Matveev _cPrime_ !

Comment

Log in or Sign up to comment
TAGS

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you