Jira is not starting

[Sanu Kuruvilla Raju]

Hi,

We have a self hosted instance of Jira (JIRA version = 7.4.0, Java Version = 1.8.0_102, Linux, 3.10.0-042stab125.5, started under root)

From last day, it suddenly stopped working. We tried to stop and start using stop-jira.sh/start-jira.sh. But we got an error saying something like 'the process with the PID is not found", 

So, we removed /opt/atlassian/jira/work/catalina.pid

Again we tried to restart, now it started to say, catalina.out is not accessible. So, we removed catalina.out also. 

Now it started successfully. When we try to access it , it is forwarded to a error.jsp file with 503 error. We checked the catalina.out, and found this exception

[c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.
com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/jira/plugins/.osgi-plugins/felix
at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.validate(DefaultOsgiPersistentCache.java:81)
at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.<init>(DefaultOsgiPersistentCache.java:38)
at com.atlassian.jira.plugin.PluginPath$JiraHomeAdapter.<init>(PluginPath.java:135)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
... 1 filtered
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.picocontainer.injectors.AbstractInjector.newInstance(AbstractInjector.java:145)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:342)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:677)
at com.atlassian.jira.component.CachingMutablePicoContainer.getComponent(CachingMutablePicoContainer.java:137)
at com.atlassian.jira.ComponentManager.getComponent(ComponentManager.java:546)
at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:194)
at com.atlassian.jira.ComponentManager.start(ComponentManager.java:165)
at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:47)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:154)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:304)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:199)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:146)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:105)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:103)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:157)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to list contents of /var/atlassian/application-data/jira/plugins/.osgi-plugins/felix/felix-cache
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1393)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399)
at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.validate(DefaultOsgiPersistentCache.java:79)

Searched the google for this error, and found this link - https://confluence.atlassian.com/jirakb/jira-does-not-start-due-to-unable-to-clean-the-cache-directory-223219163.html

So, we removed /var/atlassian/application-data/jira/plugins/.osgi-plugins/felix and restarted the server. 

It started successfully, but when we check the issues in browser, it is displaying no issues. Only our project exists now. All the issues are erased, it seems.

Any idea how we can solve this error?

[Nic Brough -Adaptavist- · Accepted solution]

This is a common error seen when you restart Confluence as the wrong user (it will work fine when you do this), then restart it again as the correct one.  It fails because the wrong user has taken ownership of files and directories such as the plugins and logs, and clobbered the permissions so that the correct owner can no longer read/write them.

So, I suspect that someone started it as root a while ago, and your restart broke when you next restarted it as the Confluence user.

[Andy Heinzer]

Good call, Nic.  The same is true of Jira.  If other admins encounter this kind of problem in the future, it is recommended to check on the user account starting Jira and make sure that user has permissions on both the installation and home folders.  For Jira on Linux installs we have this KB to help guide on to correct this kind of problem: How to fix JIRA Directories permission in Linux

[Kurasi Vishnu Priya]

Caesium-1-2 ERROR [c.a.jira.startup.IndexRecoveryLauncher] Issue index is out of date with the database by more than 24 hours. Automatic recovery is not attempted

Getting this error after the jira server upgrade from 8.4 to 8.7..

may i know why am facing this error.

Thanks

Vishnu Priya K

[Sanu Kuruvilla Raju · Accepted solution]

There was permission issue (we dont know what happened suddenly, it was running perfectly till yesterday)

We followed the instructions mentioned in this doc,

https://confluence.atlassian.com/jirakb/jira-applications-generate-a-filenotfoundexception-due-to-permission-denied-300810426.html

Again restarted. Again, the issues where not displayed, we could take the issues individually using direct URLs, but not through any filters. So, we did a integrity checking through admin panel. 

Then re-indexed the database using admin panel.

Now everything works fine.

Dont know what triggered these errors.