Jira is not starting

Question

Hi, We have a self hosted instance of Jira (JIRA version = 7.4.0, Java Version = 1.8.0_102, Linux, 3.10.0-042stab125.5, started under root) From last day, it suddenly stopped working. We tried to stop and start using stop-jira.sh/start-jira.sh. But we got an error saying something like 'the process with the PID is not found", So, we removed /opt/atlassian/jira/work/catalina.pid Again we tried to restart, now it started to say, catalina.out is not accessible. So, we removed catalina.out also. Now it started successfully. When we try to access it , it is forwarded to a error.jsp file with 503 error. We checked the catalina.out, and found this exception [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked. com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/jira/plugins/.osgi-plugins/felix at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.validate(DefaultOsgiPersistentCache.java:81) at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.(DefaultOsgiPersistentCache.java:38) at com.atlassian.jira.plugin.PluginPath$JiraHomeAdapter.(PluginPath.java:135) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ... 1 filtered at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.picocontainer.injectors.AbstractInjector.newInstance(AbstractInjector.java:145) at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:342) at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270) at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40) at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631) at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118) at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136) at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78) at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309) at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335) at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270) at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40) at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631) at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118) at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136) at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78) at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309) at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335) at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270) at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40) at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631) at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118) at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136) at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78) at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309) at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335) at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270) at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91) at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64) at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40) at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646) at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:677) at com.atlassian.jira.component.CachingMutablePicoContainer.getComponent(CachingMutablePicoContainer.java:137) at com.atlassian.jira.ComponentManager.getComponent(ComponentManager.java:546) at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:194) at com.atlassian.jira.ComponentManager.start(ComponentManager.java:165) at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:47) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:154) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:304) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:199) at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:146) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:105) at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31) at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:103) at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:157) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Failed to list contents of /var/atlassian/application-data/jira/plugins/.osgi-plugins/felix/felix-cache at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1393) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331) at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910) at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399) at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.validate(DefaultOsgiPersistentCache.java:79) Searched the google for this error, and found this link - https://confluence.atlassian.com/jirakb/jira-does-not-start-due-to-unable-to-clean-the-cache-directory-223219163.html So, we removed /var/atlassian/application-data/jira/plugins/.osgi-plugins/felix and restarted the server. It started successfully, but when we check the issues in browser, it is displaying no issues. Only our project exists now. All the issues are erased, it seems. Any idea how we can solve this error?

Sanu Kuruvilla Raju · Accepted Answer

There was permission issue (we dont know what happened suddenly, it was running perfectly till yesterday)

We followed the instructions mentioned in this doc,

https://confluence.atlassian.com/jirakb/jira-applications-generate-a-filenotfoundexception-due-to-permission-denied-300810426.html

Again restarted. Again, the issues where not displayed, we could take the issues individually using direct URLs, but not through any filters. So, we did a integrity checking through admin panel.

Then re-indexed the database using admin panel.

Now everything works fine.

Dont know what triggered these errors.

Nic Brough -Adaptavist- · Accepted Answer

This is a common error seen when you restart Confluence as the wrong user (it will work fine when you do this), then restart it again as the correct one.  It fails because the wrong user has taken ownership of files and directories such as the plugins and logs, and clobbered the permissions so that the correct owner can no longer read/write them.

So, I suspect that someone started it as root a while ago, and your restart broke when you next restarted it as the Confluence user.