Jira fails to start with lots of "Cannot execute atlassian-spring-scanner-runtime" exceptions

Hi,

I have a Jira 7.6.1 instance running on windows, and most of the time it fails to start and generates lots of the following exceptions for various plugins, then sits there waiting for plugins to finish loading until it times out. Occasionally it doesn't generate these errors and loads the plugins with no problem, but I'm struggling to find a pattern. It seems to fail more often if I start it as a service, rather than from the command line, but it isn't always the case. The log file before this point looks the same whether it is going to succeed or fail.

Any ideas?


2017-12-11 00:11:48,810 JIRA-Bootstrap INFO      [c.a.plugin.manager.DefaultPluginManager] Plugin system earlyStartup begun
2017-12-11 00:12:02,061 ThreadPoolAsyncTaskExecutor::Thread 4 ERROR      [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.atlassian.jira.jira-project-config-plugin'
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [bundle://71.0:0/META-INF/spring/spring-scanner.xml]; nested exception is java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
    at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
    at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
    at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
    at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:170)
    at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:140)
    at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
    at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:242)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:220)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:157)
    at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:207)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
    at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.checkScannerRuntimeIsNotEmbeddedInBundle(AtlassianScannerBeanDefinitionParser.java:198)
    at com.atlassian.plugin.spring.scanner.runtime.impl.AtlassianScannerBeanDefinitionParser.parse(AtlassianScannerBeanDefinitionParser.java:60)
    at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:74)
    at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1411)
    at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
    at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
    at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
    ... 20 more

 

Thanks,

Richard

1 answer

0 vote

If Jira's plugins are sometimes timing out when trying to start up, it tends to be because the system is under resourced for all the load the plugins in Jira are putting on the JVM.  I would be interested to learn more details about your specific environment, such as JVM Xmx value, the total system memory, the number of CPU cores and their speed, as well as the number of issues in Jira, and details like the plugins being loaded by Jira.   It could also be that other applications running on the same system, like a database instance or other applications like Confluence could be using up resources at the time Jira is trying to start up which might be a factor here.

From there I would want to compare your system against our JIRA Sizing Guide - Atlassian Documentation This is one way to understand if you need to Increasing memory in JIRA.   You could also follow the work-around in the KB JIRA applications System Plugin Timeout While Waiting for Add-ons to Enable.  That KB has a JVM argument you can add to Jira to tell the system to wait longer in order to load all the system plugins necessary to make Jira work.

It's also possible that a system plugin can fail to start up a number of times in Jira and as a result it can get flagged in the database as a marker to disable this plugin.  If that happens then you would need to remove the entry that corresponds to this plugin in the pluginstate table.  You can see what plugins are currently disabled in this Jira instance by running the command:

select * from pluginstate;

If any entries are listed here as system plugins and are disable, then you would need to clear those entries from the table and restart Jira once more to get these system plugins to load on startup.

Hi Andrew,

Thanks for the reply. It is a tiny system - 5 users, ~300 issues. It is running in a Windows 2016 VM on a 4-core Xeon E3-1225 v3 with 12GBs of RAM shared between a couple of other VMs that aren't really doing anything at all. CPU usage is stable at <10% and RAM is < 50% until Jira startup begins.

Whilst the plugins do timeout when it fails, all the exceptions seem to happen right at the start of the plugin load. When it does succeed then all the plugins load in <20 seconds, so I don't think performance is an issue here.

I tried increasing the Xmx value to 1024, but this didn't help, and neither did increasing the plugin wait timeout as I'm pretty certain they've failed and are never going to start.

I checked the pluginstate table, but it appeared to be empty. Is this what you'd expect?

If you've got any other ideas, I'd be very keen to hear them.

I've got full log files for a failed and successful startup, which I guess might be useful. I tried pasting them into a codeblock here, but I got an error when I hit reply saying it wasn't a valid message. Is there an easy way to share them?

Many thanks,

Richard

Actually, I thought I'd just stick the logs in pastebin:

1st part of bad log: https://pastebin.com/2Rx26yMZ
2nd part of bad log: https://pastebin.com/uSdWk22B

Good log: https://pastebin.com/30d1Ay1C

The good log contains a list of the plugins.

Thanks,

Richard

Hi Richard,

Thanks for these logs. It's very strange that this would sometimes work and other times not. It isn't clear to me exactly what would cause that behavior just yet. Did this problem start just after the most recent upgrade to 7.6.1? Or was this problem in existence before the upgrade?

I did also notice in your logs this warning:

2017-12-10 23:28:44,976 JIRA-Bootstrap WARN      [c.a.jira.health.HealthChecks] Your database is using an unsupported collation
2017-12-10 23:28:44,976 JIRA-Bootstrap WARN      [c.a.jira.health.HealthChecks] Your postgres72 database is currently using an unsupported collation: English_United States.1252. You should change this to a supported collation:  - POSIX.UTF-8
        - C.UTF-8
        - C
        - POSIX
   
    Review our documentation for more information on supported collations.  

I have seen all kinds of strange malfunctions in Jira for platforms using unsupported collations/encoding. It might not be a cause here, but I think it is worth ruling this out by fixing this collation encoding problem with these steps:

  1. Select a database type/version from our Supported platforms - Atlassian Documentation
  2. Create a new blank database per Connecting Jira applications to PostgreSQL, (paying close attention to the correct collation for that database type)
  3. Gather an XML backup of your JIRA instance. Even if you can't start JIRA, you can still find a recent one of these backups in the filesystem of the JIRA server. By default this is saved in <jira-home>/export/ folder
  4. Stop JIRA,
  5. Run the JIRA Configuration tool to tell JIRA to use the new blank database, save these changes, (if you can't run this, then you can just directly edit the <jira-home>/dbconfig.xml file to make these changes)
  6. Start JIRA up again (when JIRA starts with an empty database, it automatically launches the setup wizard)
  7. Copy the export XML zip file from <jira-home>/export/ into <jira-home>/import/
  8. Then you can import the backup using the 'Import your data' link in the setup wizard

By following these steps you can then import your previous data into a supported database with the correct database setup.

Please try these steps and let me know if you continue to see this kind of startup problem.
Regards,
Andy

Was there a solution to this? The collation shouldn't be the problem as it is correct in my case. I administer some instances and I faced the exact same problem for the first time now on a Windows Server 2016 - which is my guess for the problem with JIRA (7.7.2 in my case).

Suggest an answer

Log in or Sign up to answer
Atlassian Community Anniversary

Happy Anniversary, Atlassian Community!

This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.

Read more
Community showcase
Sarah Schuster
Posted Mar 28, 2018 in Jira Software

Can a company’s culture make or break agile adoption?

Can a new-to-agile team survive and thrive in a non-agile culture? If so, what advice would you give to those trying to be agile in a non-agile culture? What's the key(s) to success? Share your thoug...

12,054 views 15 13
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you