Jira won't start after server reboot HTTP ERROR 503

[Strong Data]

Jira won't start after server reboot. Getting these errors in the log file.

2017-12-06 10:34:23,436 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is configured. Now running Database Checklist Launcher
2017-12-06 10:34:23,436 JIRA-Bootstrap INFO [c.a.jira.startup.DatabaseChecklistLauncher] JIRA database startup checks completed successfully.
2017-12-06 10:34:23,452 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is configured. Now running Post database-configuration launchers
2017-12-06 10:34:25,631 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from C:\Program Files\Atlassian\Application Data\JIRA\dbconfig.xml
2017-12-06 10:34:26,113 JIRA-Bootstrap WARN [c.a.j.config.properties.ApplicationPropertiesStore] Exception getting property 'jira.extension.container.provider' from database. Using default
2017-12-06 10:34:26,284 JIRA-Bootstrap ERROR [c.a.jira.startup.ComponentContainerLauncher] A fatal error occurred during initialisation. JIRA has been locked.
com.atlassian.jira.exception.DataAccessException: org.ofbiz.core.entity.GenericDataSourceException: SQL Exception while executing the following:SELECT ID, user_key, lower_user_name FROM PUBLIC.app_user (The database has been closed [90098-185])
at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:401)
at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findListIteratorByCondition(WrappingOfBizDelegator.java:278)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:227)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.asList(SelectQueryImpl.java:164)
at com.atlassian.jira.user.util.UserKeyStoreImpl.buildCache(UserKeyStoreImpl.java:179)
at com.atlassian.jira.user.util.UserKeyStoreImpl.<init>(UserKeyStoreImpl.java:59)
... 3 filtered
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.picocontainer.injectors.AbstractInjector.newInstance(AbstractInjector.java:145)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:342)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getLocalInstance(DefaultPicoContainer.java:605)
at org.picocontainer.DefaultPicoContainer.getComponents(DefaultPicoContainer.java:586)
at com.atlassian.jira.component.CachingMutablePicoContainer.getComponents(CachingMutablePicoContainer.java:191)
at com.atlassian.jira.ComponentManager.runInitializingComponents(ComponentManager.java:290)
at com.atlassian.jira.ComponentManager.initialise(ComponentManager.java:149)
at com.atlassian.jira.ComponentManager.initialise(ComponentManager.java:157)
at com.atlassian.jira.startup.ComponentContainerLauncher.populateFullPicoContainer(ComponentContainerLauncher.java:48)
at com.atlassian.jira.startup.ComponentContainerLauncher.start(ComponentContainerLauncher.java:28)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:149)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:304)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:199)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:146)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:105)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:103)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:157)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.ofbiz.core.entity.GenericDataSourceException: SQL Exception while executing the following:SELECT ID, user_key, lower_user_name FROM PUBLIC.app_user (The database has been closed [90098-185])
at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:533)
at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:877)
at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:857)
at org.ofbiz.core.entity.GenericHelperDAO.findListIteratorByCondition(GenericHelperDAO.java:216)
at org.ofbiz.core.entity.GenericDelegator.findListIteratorByCondition(GenericDelegator.java:1243)
... 76 more
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed [90098-185]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.engine.Database.checkPowerOff(Database.java:496)
at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:108)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:527)
... 81 more
2017-12-06 10:34:26,315 JIRA-Bootstrap ERROR [c.a.jira.health.HealthChecks] We couldn't start JIRA
2017-12-06 10:34:26,315 JIRA-Bootstrap ERROR [c.a.jira.health.HealthChecks] An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.
See our documentation for more information on contacting our support team and creating a support zip.
2017-12-06 10:34:26,315 JIRA-Bootstrap WARN [c.a.jira.startup.DatabaseLauncher]

Please help :(

[Strong Data · Accepted solution]

So I got it all fixed up after. I figured I would post again if anyone else had the same problem.

TL;DR - H2 database corrupt?, recovered from auto backup 

So it seems like something happened to the internal H2 Database that whenever it would try to access data from it, it would get the error:

(The database has been closed [90098-185])

So I copied the h2db.mv.db and h2db.trace.db files to my computer and I figured I would try to use something like SQuirreL to copy the data to our existing SQL server. But when I would try to copy table data from H2 to SQL it gave me the same error:

(The database has been closed [90098-185])

So while I was looking through the Jira home directory I went into the 'Export' folder and lo and behold, I found all the backup .zip files that Jira had made. I didn't know that Jira automatically made backups in this folder. So I deleted the 'dbconfig.xml' file from the home directory, started Jira back up and went through the process of manually starting a new instance of Jira and setting up the db on our SQL server this time. So then I clicked the link to recover from the backup .zip file and BAM! back up and running again.

[somethingblue]

Hi Jeff,

From the error you're getting it appears this is related to the H2 database.  I did some research into this error and it could be occurring due to the JDK Version you're using or the JVM could have run out of memory to name a few reasons.

At this point I would take a look at the logs and see if you have any OutOfMemory errors.  If not, the next step is going to be to move from the internal H2 database to a Supported Database.

Cheers,

Branden

[Strong Data]

Hi Branden,

Thanks for your response. I had a quick look at the logs but I couldn't see any errors relating to being out of memory.

I'm all for moving the the data from the H2 db to our SQL server but the only thing is I can't back it up using JIRA's XML backup because I can't get it started (hindsight).  Is there anyway I can transfer the data from the db files to a SQL server?

Thanks a million.

Jeff