Plugin startup errors after update UPM

[Vitaly Razin]

After the plug-in (UPM version 2.22.2 to UPM 2.22.5) is updated, plug-ins do not start:

1. Atlassian JIRA - Plugins - Development Integration Plugin
2. JIRA DVCS Connector Plugin
3. JIRA Software Application (JIRA Software is licensed but not currently installed), JIRA Agile (UPGRADE REQUIRED).
4. Structure v4.2.0

All the recommendations were fulfilled.
The JVM parameter {{-Datlassian.plugins.enable.wait=300 }}, clean cache plugins and other recommendations.

What is done:
1. Rollback UPM to version 2.22.4.
2. Rollback UPM to version 2.22.2.
3. Removing and installing the plug-in (Structure).
Did not help.

Logs:

2017-10-02 08:58:10,453 Spring DM Context Creation Timer WARN      [o.e.g.b.e.i.dependencies.startup.DependencyWaiterApplicationContextExecutor] Timeout occurred before finding service dependencies for [NonValidatingOsgiBundleXmlApplicat
ionContext(bundle=com.almworks.jira.structure, config=osgibundle:/META-INF/spring/*.xml)]
2017-10-02 08:58:10,458 Spring DM Context Creation Timer ERROR      [c.a.p.osgi.factory.OsgiPlugin] Plugin 'com.almworks.jira.structure' never resolved service '&userProfileAccessor' with filter '(objectClass=com.atlassian.streams.spi.Us
erProfileAccessor)'
2017-10-02 08:58:10,458 Spring DM Context Creation Timer ERROR      [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.almworks.jira.structure'
org.springframework.context.ApplicationContextException: Application context initialization for 'com.almworks.jira.structure' has timed out waiting for (objectClass=com.atlassian.streams.spi.UserProfileAccessor)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:489)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:54)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:109)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
2017-10-02 08:58:10,501 Spring DM Context Creation Timer ERROR      [o.e.g.b.e.internal.support.ExtenderConfiguration] Application context refresh failed (NonValidatingOsgiBundleXmlApplicationContext(bundle=com.almworks.jira.structure, config=osgibundle:/META-INF/spring/*.xml))
org.springframework.context.ApplicationContextException: Application context initialization for 'com.almworks.jira.structure' has timed out waiting for (objectClass=com.atlassian.streams.spi.UserProfileAccessor)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:489)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:54)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:109)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
2017-10-02 08:58:10,502 Spring DM Context Creation Timer ERROR      [o.e.g.b.e.i.dependencies.startup.DependencyWaiterApplicationContextExecutor] Unable to create application context for [com.almworks.jira.structure], unsatisfied dependencies: Dependency on [(objectClass=com.atlassian.streams.spi.UserProfileAccessor)] (from bean [&userProfileAccessor])
org.springframework.context.ApplicationContextException: Application context initialization for 'com.almworks.jira.structure' has timed out waiting for (objectClass=com.atlassian.streams.spi.UserProfileAccessor)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:489)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:54)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:109)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

[Andy Heinzer · Accepted solution]

Hi Vitaly,

Sorry to hear that you are seeing a problem with loading plugins after upgrading the UPM.   Given these errors, it appears that the system is timing out trying to load dependencies.   Since you already extended the timeout period and this is still happening, then there must be something else that is causing this to happen.

I suspect that either the system is severely under resourced for the size of Jira, OR it's possible there is some other process on the system that is locking up resources at inopportune times (like Antivirus scanners can do).   The KB article Plugins are disabled at startup and various timeout errors appear in the logs after startup

 also explains how this can happen on a Confluence instance.  The steps to check for if running antivirus are the same for Jira.   However if the plugin itself has been flagged as disabled in the database, then you would need to run different SQL query to check for this in a Jira database:

select * from pluginstate;

This will show all the plugins in your Jira instance that have been disabled. 

Moving forward, I first recommend checking for antivirus as a potential cause here and excluding the jirahome and jirainstall directories from scans.

If the problem persists, then please let me know the results of that SQL query above.  In addition to this, I'd like to learn more about how many issues are in this Jira instance, how much system memory exists on this server, and how much memory the JVM Jira is using is allocated for.  One way to gather this information is to generate a support zip.  You can then open the application-properties\application.xml file.  In here you can search for the terms '-xmx' and 'database-statistics' in order to find out what your max memory for Jira is, and how many issues/custom fields/comments you have in your Jira instance.

I'd recommend allocating resources for your system based on our Jira Sizing Guide - Atlassian Documentation

[Vitaly Razin]

Problem solved. These plugins were not listed

select * from pluginstate;

they depended on another plugin. who was on this list. (com.atlassian.streams.streams-jira-plugin)