I have a Jira 7.6.1 instance running on windows, and most of the time it fails to start and generates lots of the following exceptions for various plugins, then sits there waiting for plugins to finish loading until it times out. Occasionally it doesn't generate these errors and loads the plugins with no problem, but I'm struggling to find a pattern. It seems to fail more often if I start it as a service, rather than from the command line, but it isn't always the case. The log file before this point looks the same whether it is going to succeed or fail.
2017-12-11 00:11:48,810 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system earlyStartup begun
2017-12-11 00:12:02,061 ThreadPoolAsyncTaskExecutor::Thread 4 ERROR [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.atlassian.jira.jira-project-config-plugin'
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [bundle://71.0:0/META-INF/spring/spring-scanner.xml]; nested exception is java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes, perhaps embedded inside the target plugin 'com.atlassian.jira.admin-project-config-plugin'; embedding scanner-annotations is not supported since scanner version 2.0. Use 'mvn dependency:tree' and ensure the atlassian-spring-scanner-annotation dependency in your plugin has <scope>provided</scope>, not 'runtime' or 'compile', and you have NO dependency on atlassian-spring-scanner-runtime.
... 20 more
If Jira's plugins are sometimes timing out when trying to start up, it tends to be because the system is under resourced for all the load the plugins in Jira are putting on the JVM. I would be interested to learn more details about your specific environment, such as JVM Xmx value, the total system memory, the number of CPU cores and their speed, as well as the number of issues in Jira, and details like the plugins being loaded by Jira. It could also be that other applications running on the same system, like a database instance or other applications like Confluence could be using up resources at the time Jira is trying to start up which might be a factor here.
From there I would want to compare your system against our JIRA Sizing Guide - Atlassian Documentation This is one way to understand if you need to Increasing memory in JIRA. You could also follow the work-around in the KB JIRA applications System Plugin Timeout While Waiting for Add-ons to Enable. That KB has a JVM argument you can add to Jira to tell the system to wait longer in order to load all the system plugins necessary to make Jira work.
It's also possible that a system plugin can fail to start up a number of times in Jira and as a result it can get flagged in the database as a marker to disable this plugin. If that happens then you would need to remove the entry that corresponds to this plugin in the pluginstate table. You can see what plugins are currently disabled in this Jira instance by running the command:
select * from pluginstate;
If any entries are listed here as system plugins and are disable, then you would need to clear those entries from the table and restart Jira once more to get these system plugins to load on startup.
Thanks for the reply. It is a tiny system - 5 users, ~300 issues. It is running in a Windows 2016 VM on a 4-core Xeon E3-1225 v3 with 12GBs of RAM shared between a couple of other VMs that aren't really doing anything at all. CPU usage is stable at <10% and RAM is < 50% until Jira startup begins.
Whilst the plugins do timeout when it fails, all the exceptions seem to happen right at the start of the plugin load. When it does succeed then all the plugins load in <20 seconds, so I don't think performance is an issue here.
I tried increasing the Xmx value to 1024, but this didn't help, and neither did increasing the plugin wait timeout as I'm pretty certain they've failed and are never going to start.
I checked the pluginstate table, but it appeared to be empty. Is this what you'd expect?
If you've got any other ideas, I'd be very keen to hear them.
I've got full log files for a failed and successful startup, which I guess might be useful. I tried pasting them into a codeblock here, but I got an error when I hit reply saying it wasn't a valid message. Is there an easy way to share them?
Thanks for these logs. It's very strange that this would sometimes work and other times not. It isn't clear to me exactly what would cause that behavior just yet. Did this problem start just after the most recent upgrade to 7.6.1? Or was this problem in existence before the upgrade?
I did also notice in your logs this warning:
2017-12-10 23:28:44,976 JIRA-Bootstrap WARN [c.a.jira.health.HealthChecks] Your database is using an unsupported collation 2017-12-10 23:28:44,976 JIRA-Bootstrap WARN [c.a.jira.health.HealthChecks] Your postgres72 database is currently using an unsupported collation: English_United States.1252. You should change this to a supported collation: - POSIX.UTF-8 - C.UTF-8 - C - POSIX Review our documentation for more information on supported collations.
I have seen all kinds of strange malfunctions in Jira for platforms using unsupported collations/encoding. It might not be a cause here, but I think it is worth ruling this out by fixing this collation encoding problem with these steps:
By following these steps you can then import your previous data into a supported database with the correct database setup.
Please try these steps and let me know if you continue to see this kind of startup problem.
Was there a solution to this? The collation shouldn't be the problem as it is correct in my case. I administer some instances and I faced the exact same problem for the first time now on a Windows Server 2016 - which is my guess for the problem with JIRA (7.7.2 in my case).
This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.Read more
Can a new-to-agile team survive and thrive in a non-agile culture? If so, what advice would you give to those trying to be agile in a non-agile culture? What's the key(s) to success? Share your thoug...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs