Jira Data Center startup Issue after OpenJDK 8 switch

[Jay Kantaria]

Hello,

We have been running Jira 7.13.5 data center version on RHEL two nodes cluster using Oracle Java/JRE.  Recently we wanted to give more JVM memory which is not possible with 32 bit Java, so decided to update Java to Open JDK 8.

This is causing Jira to start with multiple issues in the logs.  Some of the key errors are provided below.  Any help on getting this resolved would be appreciated.  

These errors are in order they are presented in atlassina-jira.log file when I start Jira with OpenDJK 8.

2020-03-30 16:28:18,962 ThreadPoolAsyncTaskExecutor::Thread 23 ERROR [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.atlassian.analytics.analytics-client'
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'productEventListener' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 4 of type [com.atlassian.analytics.client.logger.AnalyticsLogger]: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)
@

2020-03-30 16:28:28,999 ThreadPoolAsyncTaskExecutor::Thread 23 ERROR [o.e.g.b.e.i.dependencies.startup.DependencyWaiterApplicationContextExecutor] Unable to create application context for [com.atlassian.analytics.client], unsatisfied dependencies: none
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'productEventListener' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 4 of type [com.atlassian.analytics.client.logger.AnalyticsLogger]: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

2020-03-30 16:28:59,587 localhost-startStop-1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.kintosoft.jira.subversion-plus]
2020-03-30 16:29:00,303 active-objects-init-JiraTenantImpl{id='system'}-0 DEBUG anonymous [c.a.activeobjects.osgi.TenantAwareActiveObjects] bundle [com.kintosoft.jira.subversion-plus] created ActiveObjects
2020-03-30 16:29:00,380 localhost-startStop-1 ERROR [c.k.jira.jdbc.Jdbc4JqlDatasourceProvider] java.lang.NullPointerException
2020-03-30 16:29:00,529 localhost-startStop-1 ERROR [c.k.jira.jdbc.Jdbc4JqlDatasourceProvider] java.lang.NullPointerException
2020-03-30 16:29:00,530 localhost-startStop-1 ERROR [c.k.j.p.e.subversion.reporting.Sql4JiraMgr]
2020-03-30 16:29:00,819 localhost-startStop-1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.plugins.webhooks.jira-webhooks-plugin]
2020-03-30 16:29:00,978 localhost-startStop-1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jpo]

********************************************************************************************************************************************************************************************************
___ FAILED PLUGIN REPORT _____________________

1 plugin failed to load during JIRA startup.

'com.atlassian.analytics.analytics-client' - 'Analytics Client Plugin' failed to load.
Error creating bean with name 'productEventListener' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 4 of type [com.atlassian.analytics.client.logger.AnalyticsLogger]: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException
Error creating bean with name 'analyticsLogger' defined in URL [bundle://19.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException
Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException
null

It was loaded from /opt/atlassian/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/analytics-client-5.6.2.jar

********************************************************************************************************************************************************************************************************

2020-03-30 16:28:59,587 localhost-startStop-1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.kintosoft.jira.subversion-plus]
2020-03-30 16:29:00,303 active-objects-init-JiraTenantImpl{id='system'}-0 DEBUG anonymous [c.a.activeobjects.osgi.TenantAwareActiveObjects] bundle [com.kintosoft.jira.subversion-plus] created ActiveObjects
2020-03-30 16:29:00,380 localhost-startStop-1 ERROR [c.k.jira.jdbc.Jdbc4JqlDatasourceProvider] java.lang.NullPointerException
2020-03-30 16:29:00,529 localhost-startStop-1 ERROR [c.k.jira.jdbc.Jdbc4JqlDatasourceProvider] java.lang.NullPointerException
2020-03-30 16:29:00,530 localhost-startStop-1 ERROR [c.k.j.p.e.subversion.reporting.Sql4JiraMgr]
2020-03-30 16:29:00,819 localhost-startStop-1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.plugins.webhooks.jira-webhooks-plugin]
2020-03-30 16:29:00,978 localhost-startStop-1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jpo]

2020-03-30 16:29:04,113 Caesium-1-2 WARN [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=data-provider-cleanup; runDetails=RunDetailsImpl[startTime=1585585743826,runOutcome=SUCCESS,durationInMillis=287,message=]
2020-03-30 16:29:04,178 Caesium-1-3 WARN [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.cache.EhCacheCompactor; runDetails=RunDetailsImpl[startTime=1585585743863,runOutcome=SUCCESS,durationInMillis=315,message=Purged 0 expired entries from a total of 1,815 entries in 22 caches.]
2020-03-30 16:29:04,395 localhost-startStop-1 ERROR [c.k.j.jdbc.api.JDBC4JQLImpl] Wrong user name or password [28000-196]
2020-03-30 16:29:04,909 Caesium-1-1 WARN [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.kintosoft.jira.subversion-plus.Subversion.ScannerJob; runDetails=RunDetailsImpl[startTime=1585585743878,runOutcome=SUCCESS,durationInMillis=1031,message=success]
2020-03-30 16:29:05,260 localhost-startStop-1 ERROR [c.k.jira.jdbc.Jdbc4JqlDatasourceProvider] org.h2.jdbc.JdbcSQLException: Wrong user name or password [28000-196]

2020-03-30 16:29:08,021 Gemini Blueprint context shutdown thread2 ERROR [c.k.jira.jdbc.Jdbc4JqlDatasourceProvider] org.h2.jdbc.JdbcSQLException: Wrong user name or password [28000-196]
2020-03-30 16:29:09,084 Gemini Blueprint context shutdown thread2 ERROR [c.k.jira.jdbc.Jdbc4JqlDatasourceProvider] org.h2.jdbc.JdbcSQLException: Wrong user name or password [28000-196]
2020-03-30 16:29:11,155 Gemini Blueprint context shutdown thread2 ERROR [c.k.j.jdbc.api.JDBC4JQLImpl] Wrong user name or password [28000-196]
2020-03-30 16:29:11,158 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.kintosoft.jira.subversion-plus]
2020-03-30 16:29:11,367 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.kintosot.jira.jdbc4jql]
2020-03-30 16:29:11,792 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.keplerrominfo.jira.plugins.commons]
2020-03-30 16:29:11,893 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.kanoah.test-manager]
2020-03-30 16:29:12,109 Gemini Blueprint context shutdown thread1 ERROR [c.a.analytics.utils.JiraAnalyticsUtils] Error obtaining infoHost
2020-03-30 16:29:12,904 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.plugins.atlassian-whitelist-core-plugin]
2020-03-30 16:29:13,329 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.labs.hipchat.hipchat-for-jira-plugin]
2020-03-30 16:29:13,930 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.radiantminds.roadmaps-jira]

[Matt Doar]

Probably need to clean the work directory.

Also go to 64 bit Oracle JDK first, then move to OpenJDK. Only change one thing at a time