JIRA Fails to Start Due to Bundled Plugin is Disabled After Upgrade from 4.4 to 5.0

[Jeffrey Toce]

I'm running JIRA on Amazon EC2 - I want to both migrate JIRA to a new server and then upgrade to 5.0. Here is what I've done:

1. Export XML backup of JIRA
2. Migrated from 32-bit t1.micro instance to a new 64-bit m1.small instance
3. Installed JIRA 4.4 on the new server
4. Import XML backup of JIRA
   1. NOTE: JIRA was working on the new server with my data at this point
5. Upgrade JIRA from 4.4 to 5.0

When I try to launch JIRA I get the following error:

JIRA Startup Failed

You cannot access JIRA at present. Look at the table below to identify the reasons

Description The following plugins are required by JIRA, but have not been started: Atlassian JIRA - Plugins - Project Config Plugin (com.atlassian.jira.jira-project-config-plugin) Embedded Gadgets Plugin (com.atlassian.gadgets.embedded) Gadget Dashboard Plugin (com.atlassian.gadgets.dashboard) Opensocial Plugin (com.atlassian.gadgets.opensocial) Gadget Directory Plugin (com.atlassian.gadgets.directory) Atlassian JIRA - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)

I've tried these KB articles:

http://confluence.atlassian.com/display/JIRAKB/JIRA+Fails+to+Start+Due+to+Bundled+Plugin+is+Disabled

http://confluence.atlassian.com/pages/viewpage.action?pageId=212173447

And then I tried this article to increase the memory to 512m - 1024m:

http://confluence.atlassian.com/display/JIRA/Increasing+JIRA+Memory

Here is my catalina log: (catalina.log)

Any help would be greatly appreciated!

Added latest log file: (catalina.log)

[Andrew Frayling · Accepted solution]

Thanks for trying the increased memory, must be getting closer :/

For timeouts you could try: http://confluence.atlassian.com/display/CONFKB/Plugins+are+Disabled+at+Startup+and+Various+Timeout+Errors+Appear+in+the+Logs+After+Startup

It's for Confluence, but may work for JIRA too. Try setting JVM_SUPPORT_RECOMMENDED_ARGS="-Datlassian.plugins.enable.wait=300" in setenv.sh/bat

[Sorin Sbarnea (Citrix)]

Jira is not something that you can really run inder EC2 mini or micro, as it is very CPU intensive even for small installations. Still, you can find other hosting alternatives for less money than AWS, if price is a limitation.

[Harry Chan]

Hi, this has been resolved, but I'd just like to explain the possible cause. Amazon EC2 small actually has less burst CPU power than Amazon EC2 micro. JIRA startup possible requires more CPU than available on small and hence the timeout.

If running under EC2, I'd advertise looking at possible optimization etc and note that it is a single core environment in this case.

[Paul Schmitt]

JIRA Build : 5.1.3#782-sha1:4389c897ff46ac633147bfa0023fbc37f3cb8ca3

Java Version : 1.6.0_25 - Sun Microsystems Inc.

See answer:

https://answers.atlassian.com/questions/40618/atlas-run-for-sdk-3-8-and-jira-5-0-fails-with-error-in-com-atlassian-sal-jira

I was just getting the Jira eval to run I tried most of the above in this post (increasing memory, setting wait to 300), what got things working was down grading to Java 1.6

Maximum Allowable Memory : 1933MB

Total Memory : 247MB

Free Memory : 185MB

Used Memory : 62MB

atlassian.plugins.enable.wait : 300 // in the catalina.out - never counted down more than 10 sec after moving the java 1.6

[Jeffrey Toce]

• Did increasing the timeout work?

YES- that seemed to do the trick. It's not exactly running fast- but at least it's running. Thanks.

[Andrew Frayling]

Cool, glad it worked in the end. I've converted the timeout comment to an answer. If you could mark that one as correct that'd be grand.

Cheers.

[Jeffrey Toce]

Andrew- you were right. Increasing the plug-in load time worked for me in JIRA. I can't convert a comment to an answer- but if you do it, then I'll gladly mark it as the answer. Thanks!

[Andrew Frayling]

That shouldn't be a problem as JIRA being able to set PermGen to 256m is a good thing - any less and I'd expect to see:

java.lang.OutOfMemoryError: PermGen space

You could try increasing it following the instructions at http://confluence.atlassian.com/display/JIRAKB/JIRA+Crashes+Due+to+'java.lang.OutOfMemoryError+PermGen+Space+Error , but I'd expect 256m to be fine, unless you are getting the above error?

Can I recap what's been tried so far as I'm getting a bit lost:

• Increased min and max memory to 1024m
• Deleted .bundled_plugins and .osgi-plugins
• Ran SQL to find disabled plugins
• Deleted incompatible plugins from installed-plugins

What I'm not sure about is:

• Has the update SQL statement detailed at http://confluence.atlassian.com/display/JIRAKB/JIRA+Fails+to+Start+Due+to+Bundled+Plugin+is+Disabled been run?
• Is there any anti-virus running on the server?
• Did increasing the timeout work?
• Are there any other apps running on the same EC2 instance?

Cheers.

[Jeffrey Toce]

Back to the memory issue- could it be PermGen rather than the JVM? Here is what I'm getting at launch:

Detecting JVM PermGen support...

PermGen switch is supported. Setting to 256m

[Jeffrey Toce]

Set both the min and max to 1024m and still had the same problem. No love.

Although the first plugin to fail was different this time:

2012-03-19 21:57:10,632 Timer-0 WARN [internal.dependencies.startup.DependencyWaiterApplicationContextExecutor] Timeout occurred before finding service dependencies for [NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.applinks.plugin, config=osgibundle:/META-INF/spring/*.xml)]

2012-03-19 21:57:12,141 Timer-0 ERROR [internal.dependencies.startup.DependencyWaiterApplicationContextExecutor] Unable to create application context for [com.atlassian.applinks.plugin], unsatisfied dependencies: Dependency on [(objectClass=com.atlassian.sal.api.net.RequestFactory)] (from bean [&requestFactory]), Dependency on [(objectClass=com.atlassian.plugins.rest.common.json.JaxbJsonMarshaller)] (from bean [&jaxbJsonMarshaller])

org.springframework.context.ApplicationContextException: Application context initialization for 'com.atlassian.applinks.plugin' has timed out

at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)

at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)

at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)

at java.util.TimerThread.mainLoop(Unknown Source)

at java.util.TimerThread.run(Unknown Source)

Appending the log file to the original post.

[Jeffrey Toce]

512m was the min and 1024m was the max. I'll humor you gladly. Also- is there another way to increase the plugin load timeout?

[Andrew Frayling]

The times I've seen that before - https://answers.atlassian.com/questions/18327/unzipped-jira-stand-alone-initial-application-start-plugins-setup-errors-results-in-lock and https://answers.atlassian.com/questions/40922/jira-deployment-dependency-errors it's been a resource issue.

When you said:

"And then I tried this article to increase the memory to 512m - 1024m:"

what increased to 1024m? The minimum, the maximum or both? If it was just the maximum can you humour me and set both the minimum and the maximum to 1024m?

[Jeffrey Toce]

Here is the first error from the logs:

2012-03-19 21:06:02,956 Timer-0 WARN [internal.dependencies.startup.DependencyWaiterApplicationContextExecutor] Timeout occurred before finding service dependencies for [NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.activeobjects.activeobjects-plugin, config=osgibundle:/META-INF/spring/*.xml)]

2012-03-19 21:06:04,525 Timer-0 ERROR [internal.dependencies.startup.DependencyWaiterApplicationContextExecutor] Unable to create application context for [com.atlassian.activeobjects.activeobjects-plugin], unsatisfied dependencies: Dependency on [(objectClass=com.atlassian.sal.api.user.UserManager)] (from bean [&userManager]), Dependency on [(objectClass=com.atlassian.sal.api.auth.LoginUriProvider)] (from bean [&loginUriProvider]), Dependency on [(objectClass=com.atlassian.sal.api.websudo.WebSudoManager)] (from bean [&webSudoManager])

org.springframework.context.ApplicationContextException: Application context initialization for 'com.atlassian.activeobjects.activeobjects-plugin' has timed out

at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)

at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)

at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)

at java.util.TimerThread.mainLoop(Unknown Source)

at java.util.TimerThread.run(Unknown Source)

[Andrew Frayling]

Hi Jeffrey,

With the KB article at http://confluence.atlassian.com/display/JIRAKB/JIRA+Fails+to+Start+Due+to+Bundled+Plugin+is+Disabled did the SQL query return any rows? Even if it returned zero rows it's worth deleting <JIRA Home>/plugins/.bundled_plugins and <JIRA Home>/plugins/.osgi-plugins and restarting JIRA to see if that works (JIRA will recreate them on restart).

The other (random) reason for this error seems to be if there is anti-virus scanning the JIRA directories on the server or even if certain anti-virus software is present. This KB article has more details on problematic anti-virus software - http://confluence.atlassian.com/display/JIRAKB/Crashes+and+Performance+Issues+Troubleshooting

Andrew.

[Jeffrey Toce]

mysql> select * from propertyentry e join propertystring s on e.id=s.id where e.property_key like '%plugin.state%';

+-------+-----------------+-----------+-----------------------------------------------------------+--------------+-------+---------------+

| ID | ENTITY_NAME | ENTITY_ID | PROPERTY_KEY | propertytype | ID | propertyvalue |

+-------+-----------------+-----------+-----------------------------------------------------------+--------------+-------+---------------+

| 11709 | jira.properties | 1 | jira.plugin.state-.com.pyxis.greenhopper.jira | 5 | 11709 | false |

| 11710 | jira.properties | 1 | jira.plugin.state-.com.pyxis.jira.links.hierarchy.reports | 5 | 11710 | false |

+-------+-----------------+-----------+-----------------------------------------------------------+--------------+-------+---------------+

2 rows in set (0.00 sec)

Although after reading this:

http://confluence.atlassian.com/display/JIRAKB/NoClassDefFoundError%2C+ClassNotFoundException+or+NoSuchMethodException+due+to+Incompatible+Plugin

I'm in a mood to just delete all of the incompatible plugins in <JIRA_HOME>/plugins/installed-plugins.

Also- this is running on a bitnami LAMP stack on EC2- so while it is a virtualized environment, it does not have any anti-virus software installed.

[Jeffrey Toce]

Deleting the plugin directories <JIRA Home>/plugins/.bundled_plugins and <JIRA Home>/plugins/.osgi-plugins and restarting JIRA did nothing. Neither did deleting the incompatible plugins in <JIRA_HOME>/plugins/installed-plugins.

I'm still getting this in the log file:

2012-03-19 21:42:04,408 main FATAL [atlassian.jira.startup.JiraStartupLogger]

********************************************************************************************************************************************************************************************************

The following plugins are required by JIRA, but have not been started: Atlassian JIRA - Plugins - Project Config Plugin (com.atlassian.jira.jira-project-config-plugin), Embedded Gadgets Plugin (com.atlassian.gadgets.embedded), Gadget Dashboard Plugin (com.atlassian.gadgets.dashboard), Opensocial Plugin (com.atlassian.gadgets.opensocial), Gadget Directory Plugin (com.atlassian.gadgets.directory), Atlassian JIRA - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)

********************************************************************************************************************************************************************************************************