How to set up a couple of nodes of JIRA DataCenter to work with different names of node

[Aleksandr Karev]

Hi!

Is there anybody who have an installation and setup experience of JIRA DataCenter?
Can you give me an advice how to fix a problem which I run into.

There are:
1) two JIRA nodes of JIRA 7.5.4 build: 75010 (Node1, Node2 for example)
2) two JIRA nodes located of separated virtual servers: Debian GNU/Linux 9 (stretch)
3) servers in one local network (192.168.1.x/24)
4) no firewall in network and between servers
5) Shared home folder located on the Node1 (via nfs)
6) there are no load balancer(just simple JIRA nodes)

Here is info around the situation:
1) Both JIRA nodes don't work properly(can start, but a user can't log in) at one moment if they have different value of "jira.node.id" in cluster.properties file.
2) Both JIRA nodes can start and a user can log in) at one moment if they have identical value of "jira.node.id" in cluster.properties file
3) Both JIRA nodes work properly(can start and a user can log in) if they work at not the same time and have a different value of "jira.node.id" in cluster.properties file

-------

Node1 (IP: 192.168.101.30):

-------

less /etc/exports
/var/atlassian/application-data/jira-datacenter-home 192.168.101.28(rw,sync,no_subtree_check)
/var/atlassian/application-data/jira-datacenter-home 192.168.101.30(rw,sync,no_subtree_check)
/var/atlassian/application-data/jira-datacenter-home 127.0.0.1(rw,sync,no_subtree_check)

less /etc/fstab
127.0.0.1:/var/atlassian/application-data/jira-datacenter-home /mnt/jira-datacenter-home nfs user,async,noexec,dev,rw,auto 0 0

--------

Node2 (IP: 192.168.101.28):

--------

less /etc/fstab
192.168.101.30:/var/atlassian/application-data/jira-datacenter-home /mnt/jira-datacenter-home nfs user,async,noexec,dev,rw,auto 0 0

-----------------------------------------------------------------------

An affected node shows these lines in the log during the starting Jira:
-----------------------------------------------------------------------

2018-10-17 16:29:41,178 localhost-startStop-1 WARN [n.sf.ehcache.CacheManager] Cache com.atlassian.jira.task.TaskManagerImpl.taskMaprequested bootstrap but a CacheException occured. Error bootstrapping from remote peer. Message was: java.lang.reflect.InvocationTargetException
net.sf.ehcache.distribution.RemoteCacheException: Error bootstrapping from remote peer. Message was: java.lang.reflect.InvocationTargetException
... 40 more
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
... 45 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
... 49 more
2018-10-17 16:29:41,267 localhost-startStop-1 WARN [n.sf.ehcache.CacheManager] Cache com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCacherequested bootstrap but a CacheException occured. Error bootstrapping from remote peer. Message was: java.lang.reflect.InvocationTargetException
net.sf.ehcache.distribution.RemoteCacheException: Error bootstrapping from remote peer. Message was: java.lang.reflect.InvocationTargetException
at net.sf.ehcache.distribution.RMIBootstrapCacheLoader.doLoad(RMIBootstrapCacheLoader.java:176)
... 49 more
2018-10-17 16:29:41,363 localhost-startStop-1 WARN [n.sf.ehcache.CacheManager] Cache com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizGroupCache.groupCacherequested bootstrap but a CacheException occured. Error bootstrapping from remote peer. Message was: java.lang.reflect.InvocationTargetException
net.sf.ehcache.distribution.RemoteCacheException: Error bootstrapping from remote peer. Message was: java.lang.reflect.InvocationTargetException
... 49 more
2018-10-17 16:29:41,387 localhost-startStop-1 WARN [c.atlassian.jira.NoopCacheEventListenerFactory] It looks like CACHE-95 has been fixed. This workaround should be removed.
2018-10-17 16:29:42,833 localhost-startStop-1 INFO [c.a.j.c.embedded.ofbiz.OfBizUserDao] Using full user cache
2018-10-17 16:29:43,499 localhost-startStop-1 ERROR [c.a.jira.startup.ComponentContainerLauncher] A fatal error occurred during initialisation. JIRA has been locked.
java.util.concurrent.CompletionException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
... 19 more
2018-10-17 16:29:43,537 localhost-startStop-1 ERROR [c.a.jira.health.HealthChecks] We couldn't start JIRA
2018-10-17 16:29:43,538 localhost-startStop-1 ERROR [c.a.jira.health.HealthChecks] An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.
See our documentation for more information on contacting our support team and creating a support zip.
2018-10-17 16:29:43,538 localhost-startStop-1 INFO [c.a.jira.startup.DatabaseLauncher] Database transactions enabled: true
2018-10-17 16:29:43,539 localhost-startStop-1 INFO [c.a.jira.startup.DatabaseLauncher] Using JIRA's default for database transaction isolation level: 2
2018-10-17 16:29:43,551 localhost-startStop-1 INFO [c.a.jira.startup.JiraStartupLogger]
Database configuration OK
...
2018-10-17 16:29:43,946 localhost-startStop-1 ERROR [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.
com.atlassian.jira.InfrastructureException: Error occurred while starting Plugin Manager. Unable to resolve component: interface com.atlassian.plugin.SplitStartupPluginSystemLifecycle
at com.atlassian.jira.ComponentManager$PluginSystem.earlyStartup(ComponentManager.java:648)
...
at com.atlassian.jira.cluster.distribution.JiraCacheManagerPeerProvider.lambda$wrapCachePeer$4(JiraCacheManagerPeerProvider.java:112)
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused
... 15 more
Caused by: java.net.ConnectException: Connection refused
... 19 more
2018-10-17 16:29:44,595 localhost-startStop-1 ERROR [c.a.jira.health.HealthChecks] Could not execute health check, DatabaseConfigurationManager not available.
2018-10-17 16:29:44,595 localhost-startStop-1 ERROR [c.a.jira.health.HealthChecks] Could not execute health check, DatabaseConfigurationManager not available.
2018-10-17 16:29:44,596 localhost-startStop-1 ERROR [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: startup-unexpected) , Desc = We couldn't start JIRA , Exception = An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.<br/>See our documentation for more information on contacting our support team and creating a support zip.), (Event: Level = (EventLevel: fatal) , Key = (EventType: database) , Desc = Could not execute health check, DatabaseConfigurationManager not available. , Exception = ]
...

17-Oct-2018 16:29:44.953 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [http-nio-8080]
17-Oct-2018 16:29:44.953 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
17-Oct-2018 16:29:44.972 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [http-nio-8081]
17-Oct-2018 16:29:44.976 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [https-jsse-nio-8446]
17-Oct-2018 16:29:44.988 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 46767 ms

-----------------------------------------------------------------------
When I try to access to JIRA WebUI via browser these lines come up in the log in of the affected node:
-----------------------------------------------------------------------
2018-10-17 16:32:50,533 https-jsse-nio-8446-exec-4 ERROR anonymous 992x1x1 - 192.168.101.105 / [c.atlassian.instrumentation.Instrument] Unable to snapshot thread local operations (implementation of OpTimerFactory is not a ThreadLocalOpTimerFactory): null
2018-10-17 16:32:50,630 https-jsse-nio-8446-exec-5 WARN anonymous 992x2x1 - 192.168.101.105 /secure/errors.jsp [c.a.jira.security.JiraSecurityFilter] Rejecting security-sensitive request that bypassed Johnson filter: /secure/errors.jsp

[KellyW]

Hi Aleksandr,

The "jira.node.id" is a unique ID that match the username and the BalancerMember entry in the Apache configuration. Having to say that load balancer is required in the Data Center setup.

Jira Data Center relies on a load balancer to balance the traffic between the nodes, and this guide assumes that you already have one set up. You can use a load balancer of your choice, just make sure it meets these requirements:

• Supports "cookie based session affinity", also known as "sticky sessions".
• Can route HTTP/HTTPS traffic to one of the available nodes.
• Can determine whether a node is available or not, and route requests to other nodes if needed.
• All Atlassian applications and other REST clients must access your nodes through the load balancer.

Or you can just turn your proxy into a load balancer.

Many bigger installations of Jira already have a reverse proxy configured, and many reverse proxies can do load balancing as well. We've provided some examples on how to use your proxy as a load balancer. See Load balancer examples.

For more context, check out Installing Jira Data Center.

Regards,
Kelly

[Aleksandr Karev]

Hi Kelly!

Thanks for info!

Ok. "jira.node.id" is just for a load balancer, but anyway JIRA node must work properly whatever happends with the load balancer. And JIRA node must work properly with any value of this "jira.node.id".

This is the main question: Why JIRA nodes can't work properly with any value of this "jira.node.id"?