Jira fails to start with lots of "Cannot execute atlassian-spring-scanner-runtime" exceptions

Question

Hi, I have a Jira 7.6.1 instance running on windows, and most of the time it fails to start and generates lots of the following exceptions for various plugins, then sits there waiting for plugins to finish loading until it times out. Occasionally it doesn't generate these errors and loads the plugins with no problem, but I'm struggling to find a pattern. It seems to fail more often if I start it as a service, rather than from the command line, but it isn't always the case. The log file before this point looks the same whether it is going to succeed or fail. Any ideas? 2017-12-11 00:11:48,810 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system earlyStartup begun 2017-12-11 00:12:02,061 ThreadPoolAsyncTaskExecutor::Thread 4 ERROR [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.atlassian.jira.jira-project-config-plugin' org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [bundle://71.0:0/META-INF/spring/spring-scanner.xml]; nested exception is java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has provided, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime. at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414) at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336) at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304) at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181) at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217) at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188) at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:170) at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:140) at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129) at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609) at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:60) at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:242) at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85) at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:220) at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224) at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177) at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:157) at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has provided, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime. at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.checkScannerRuntimeIsNotEmbeddedInBundle(AtlassianScannerBeanDefinitionParser.java:198) at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.parse(AtlassianScannerBeanDefinitionParser.java:60) at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:74) at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1411) at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401) at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138) at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94) at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508) at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392) ... 20 more Thanks, Richard

Frank Butzek · Answer

Was there a solution to this? The collation shouldn't be the problem as it is correct in my case. I administer some instances and I faced the exact same problem for the first time now on a Windows Server 2016 - which is my guess for the problem with JIRA (7.7.2 in my case).

Andy Heinzer · Answer

Hi Richard, Thanks for these logs. It's very strange that this would sometimes work and other times not. It isn't clear to me exactly what would cause that behavior just yet. Did this problem start just after the most recent upgrade to 7.6.1? Or was this problem in existence before the upgrade? I did also notice in your logs this warning: 2017-12-10 23:28:44,976 JIRA-Bootstrap WARN [c.a.jira.health.HealthChecks] Your database is using an unsupported collation 2017-12-10 23:28:44,976 JIRA-Bootstrap WARN [c.a.jira.health.HealthChecks] Your postgres72 database is currently using an unsupported collation: English_United States.1252. You should change this to a supported collation: - POSIX.UTF-8 - C.UTF-8 - C - POSIX Review our documentation for more information on supported collations. I have seen all kinds of strange malfunctions in Jira for platforms using unsupported collations/encoding. It might not be a cause here, but I think it is worth ruling this out by fixing this collation encoding problem with these steps: * Select a database type/version from our Supported platforms - Atlassian Documentation * Create a new blank database per Connecting Jira applications to PostgreSQL, (paying close attention to the correct collation for that database type) * Gather an XML backup of your JIRA instance. Even if you can't start JIRA, you can still find a recent one of these backups in the filesystem of the JIRA server. By default this is saved in /export/ folder * Stop JIRA, * Run the JIRA Configuration tool to tell JIRA to use the new blank database, save these changes, (if you can't run this, then you can just directly edit the /dbconfig.xml file to make these changes) * Start JIRA up again (when JIRA starts with an empty database, it automatically launches the setup wizard) * Copy the export XML zip file from /export/ into /import/ * Then you can import the backup using the 'Import your data' link in the setup wizard By following these steps you can then import your previous data into a supported database with the correct database setup. Please try these steps and let me know if you continue to see this kind of startup problem. Regards, Andy

Richard Sinden_f8cft1 · Answer

Actually, I thought I'd just stick the logs in pastebin:

1st part of bad log: https://pastebin.com/2Rx26yMZ
2nd part of bad log: https://pastebin.com/uSdWk22B

Good log: https://pastebin.com/30d1Ay1C

The good log contains a list of the plugins.

Thanks,

Richard