I am facing Unexpected exception during JIRA startup. This JIRA instance will not be able to recover

[hivealbert]

com.atlassian.jira.exception.DataAccessException: org.ofbiz.core.entity.GenericDataSourceException: SQL Exception while executing the following:SELECT JOB_ID, NEXT_RUN FROM PUBLIC.clusteredjob WHERE (NEXT_RUN IS NOT NULL ) (General error: "java.lang.IllegalStateException: File corrupted in chunk 66186, expected page length =< 192, got 1969711470 [1.4.185/6]"; SQL statement: SELECT JOB_ID, NEXT_RUN FROM PUBLIC.clusteredjob WHERE (NEXT_RUN IS NOT NULL ) [50000-185]) at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:401) at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findListIteratorByCondition(WrappingOfBizDelegator.java:278) at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:227) at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.visitWith(SelectQueryImpl.java:185) at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:110) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130) at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275) at com.atlassian.jira.scheduler.JiraCaesiumSchedulerService.startImpl(JiraCaesiumSchedulerService.java:36) at com.atlassian.scheduler.core.AbstractSchedulerService.start(AbstractSchedulerService.java:180) at com.atlassian.jira.scheduler.JiraSchedulerLauncher.proceedIfAllClear(JiraSchedulerLauncher.java:36) at com.atlassian.jira.scheduler.JiraSchedulerLauncher.start(JiraSchedulerLauncher.java:22) at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:56) at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:170) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:144) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:304) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:199) at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:102) at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31) at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:100) at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154) at java.lang.Thread.run(Thread.java:745) Caused by: org.ofbiz.core.entity.GenericDataSourceException: SQL Exception while executing the following:SELECT JOB_ID, NEXT_RUN FROM PUBLIC.clusteredjob WHERE (NEXT_RUN IS NOT NULL ) (General error: "java.lang.IllegalStateException: File corrupted in chunk 66186, expected page length =< 192, got 1969711470 [1.4.185/6]"; SQL statement: SELECT JOB_ID, NEXT_RUN FROM PUBLIC.clusteredjob WHERE (NEXT_RUN IS NOT NULL ) [50000-185]) at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:533) at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:877) at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:857) at org.ofbiz.core.entity.GenericHelperDAO.findListIteratorByCondition(GenericHelperDAO.java:216) at org.ofbiz.core.entity.GenericDelegator.findListIteratorByCondition(GenericDelegator.java:1243) at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:398) ... 24 more Caused by: org.h2.jdbc.JdbcSQLException: General error: "java.lang.IllegalStateException: File corrupted in chunk 66186, expected page length =< 192, got 1969711470 [1.4.185/6]"; SQL statement: SELECT JOB_ID, NEXT_RUN FROM PUBLIC.clusteredjob WHERE (NEXT_RUN IS NOT NULL ) [50000-185] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.h2.command.Command.executeQuery(Command.java:209) at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:108) at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83) at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83) at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:527) ... 29 more Caused by: java.lang.IllegalStateException: File corrupted in chunk 66186, expected page length =< 192, got 1969711470 [1.4.185/6] at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773) at org.h2.mvstore.Page.read(Page.java:640) at org.h2.mvstore.Page.read(Page.java:194) at org.h2.mvstore.MVStore.readPage(MVStore.java:1830) at org.h2.mvstore.MVMap.readPage(MVMap.java:736) at org.h2.mvstore.Page.getChildPage(Page.java:216) at org.h2.mvstore.Cursor.fetchNext(Cursor.java:150) at org.h2.mvstore.Cursor.next(Cursor.java:50) at org.h2.mvstore.db.TransactionStore$TransactionMap$2.fetchNext(TransactionStore.java:1467) at org.h2.mvstore.db.TransactionStore$TransactionMap$2.next(TransactionStore.java:1510) at org.h2.mvstore.db.TransactionStore$TransactionMap$2.next(TransactionStore.java:1454) at org.h2.mvstore.db.MVPrimaryIndex$MVStoreCursor.next(MVPrimaryIndex.java:408) at org.h2.index.IndexCursor.next(IndexCursor.java:277) at org.h2.table.TableFilter.next(TableFilter.java:360) at org.h2.command.dml.Select.queryFlat(Select.java:532) at org.h2.command.dml.Select.queryWithoutCache(Select.java:644) at org.h2.command.dml.Query.query(Query.java:322) at org.h2.command.dml.Query.query(Query.java:290) at org.h2.command.dml.Query.query(Query.java:36) at org.h2.command.CommandContainer.query(CommandContainer.java:90) at org.h2.command.Command.executeQuery(Command.java:197)  

please suggest the steps need to be followed to resolve this error.

[Nic Brough -Adaptavist-]

Your database is corrupted.

In the short term, the fix is to install a clean development or test Jira (this is not a production system, obviously, as you're using the h2 database), and if you needed any of the data you were testing with, restore it from the last backup taken.

In the longer term, switch to a supported database - it's worth it for persistent test systems.

[hivealbert]

Thanks Nic, can you please specify, how my database got corrupted, what are the possible reasons?

[Nic Brough -Adaptavist-]

It's not particularly relevant how it got corrupted, but the most common causes I've seen for h2 failures are improper terminations of the server process (either a hard shutdown or something or someone killing it).   But it also fails on running out of disk space, running out of memory, or even just coming under significant load.  In most cases though, I've never bothered to do any real analysis, it's never been worth the effort.  Just rebuild the dev/test system.

[hivealbert]

Thanks Nic for quick help.