JIRa CORE server DOWN!!

Stefano Sbalchiero March 4, 2019

HI, my JIRa server isn't working.

I try to check but i don't undertand what is happen

 

this is the only page that i can reach via browser

jiraerror.JPGthis is the jiracore131218222258-stdout.2019-03-04 log file

 

----------------------------------------

2019-03-04 08:57:00,546 JIRA-Bootstrap DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] onPluginEnabledEvent attaching unbound <ao> to [softwareplant.bigtemplate]
2019-03-04 08:57:00,546 JIRA-Bootstrap DEBUG [c.a.activeobjects.osgi.TenantAwareActiveObjects] init bundle [com.softwareplant.bigtemplate.jiraserver]
2019-03-04 08:57:00,546 JIRA-Bootstrap DEBUG [c.a.activeobjects.osgi.TenantAwareActiveObjects] bundle [com.softwareplant.bigtemplate.jiraserver] loading new AO promise for JiraTenantImpl{id='system'}
2019-03-04 08:57:00,546 JIRA-Bootstrap DEBUG [c.a.activeobjects.osgi.TenantAwareActiveObjects] setAoConfiguration [com.softwareplant.bigtemplate.jiraserver]
2019-03-04 08:57:00,546 JIRA-Bootstrap DEBUG [c.a.activeobjects.osgi.TenantAwareActiveObjects] bundle [com.softwareplant.bigtemplate.jiraserver] got ActiveObjectsConfiguration
2019-03-04 08:57:00,546 active-objects-init-JiraTenantImpl{id='system'}-0 DEBUG anonymous [c.a.activeobjects.osgi.TenantAwareActiveObjects] bundle [com.softwareplant.bigtemplate.jiraserver] creating ActiveObjects
2019-03-04 08:57:00,624 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system earlyStartup ended
2019-03-04 08:57:00,968 JIRA-Bootstrap INFO [c.a.j.c.cache.pauser.NonClusteredReplicationPauserManager] Non-clustered mode: ReplicationPauserManager implemented by NonClusteredReplicationPauserManager
2019-03-04 08:57:01,483 JIRA-Bootstrap ERROR [c.a.j.component.pico.ComponentManager] Error occurred while starting component 'com.atlassian.jira.service.DefaultServiceManager'.
org.ofbiz.core.util.GeneralRuntimeException: Error creating GenericValue (SQL Exception while getting value: (IO Exception: "java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]" [90028-185]))
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:274)
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:51)
at com.atlassian.jira.ofbiz.DefaultOfBizListIterator$LookaheadIterator.<init>(DefaultOfBizListIterator.java:167)
at com.atlassian.jira.ofbiz.DefaultOfBizListIterator$LookaheadIterator.<init>(DefaultOfBizListIterator.java:161)
at com.atlassian.jira.ofbiz.DefaultOfBizListIterator.iterator(DefaultOfBizListIterator.java:153)
at com.atlassian.jira.ofbiz.WrappingOfBizListIterator.iterator(WrappingOfBizListIterator.java:123)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:230)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.consumeWith(SelectQueryImpl.java:214)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.singleValue(SelectQueryImpl.java:191)
at com.atlassian.jira.scheduler.OfBizClusteredJobDao.find(OfBizClusteredJobDao.java:88)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.getJobDetails(CaesiumSchedulerService.java:230)
at com.atlassian.scheduler.core.DelegatingSchedulerService.getJobDetails(DelegatingSchedulerService.java:97)
at com.atlassian.jira.service.DefaultServiceManager.ensureServiceScheduled(DefaultServiceManager.java:530)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.atlassian.jira.service.DefaultServiceManager.ensureServicesScheduled(DefaultServiceManager.java:523)
at com.atlassian.jira.service.DefaultServiceManager.start(DefaultServiceManager.java:104)
at com.atlassian.jira.component.pico.ComponentManager.runStartable(ComponentManager.java:325)
at com.atlassian.jira.component.pico.ComponentManager.quickStart(ComponentManager.java:241)
at com.atlassian.jira.component.pico.ComponentManager.start(ComponentManager.java:182)
at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:45)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:143)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:304)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:199)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:102)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:100)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.ofbiz.core.entity.GenericDataSourceException: SQL Exception while getting value: (IO Exception: "java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]" [90028-185])
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.getValue(SqlJdbcUtil.java:700)
at org.ofbiz.core.entity.EntityListIterator.currentGenericValue(EntityListIterator.java:181)
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:267)
... 37 more
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception: "java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]" [90028-185]
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:287)
at org.h2.message.DbException.toSQLException(DbException.java:268)
at org.h2.message.TraceObject.logAndConvert(TraceObject.java:352)
at org.h2.jdbc.JdbcBlob.getBytes(JdbcBlob.java:101)
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.getBlobAsByteArray(SqlJdbcUtil.java:747)
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.getValue(SqlJdbcUtil.java:695)
... 39 more
Caused by: java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]
at org.h2.mvstore.StreamStore$Stream.read(StreamStore.java:427)
at org.h2.util.IOUtils.copy(IOUtils.java:167)
at org.h2.jdbc.JdbcBlob.getBytes(JdbcBlob.java:95)
... 41 more
Caused by: java.lang.IllegalStateException: Reading from cache:nio:C:/Program Files/Atlassian/Application Data/JIRA/database/h2db.mv.db failed; file length 205557760 read length 768 at 207003968 [1.4.185/1]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:443)
at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
at org.h2.mvstore.Page.read(Page.java:189)
at org.h2.mvstore.MVStore.readPage(MVStore.java:1830)
at org.h2.mvstore.MVMap.readPage(MVMap.java:736)
at org.h2.mvstore.Page.getChildPage(Page.java:216)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:468)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:469)
at org.h2.mvstore.MVMap.get(MVMap.java:450)
at org.h2.mvstore.StreamStore.getBlock(StreamStore.java:347)
at org.h2.mvstore.StreamStore$Stream.nextBuffer(StreamStore.java:464)
at org.h2.mvstore.StreamStore$Stream.read(StreamStore.java:421)
... 43 more
Caused by: java.io.EOFException
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:431)
... 54 more
2019-03-04 08:57:01,483 JIRA-Bootstrap ERROR [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.
com.atlassian.jira.InfrastructureException: Error occurred while starting component 'com.atlassian.jira.service.DefaultServiceManager'.
at com.atlassian.jira.component.pico.ComponentManager.runStartable(ComponentManager.java:329)
at com.atlassian.jira.component.pico.ComponentManager.quickStart(ComponentManager.java:241)
at com.atlassian.jira.component.pico.ComponentManager.start(ComponentManager.java:182)
at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:45)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:143)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:304)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:199)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:102)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:100)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.ofbiz.core.util.GeneralRuntimeException: Error creating GenericValue (SQL Exception while getting value: (IO Exception: "java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]" [90028-185]))
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:274)
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:51)
at com.atlassian.jira.ofbiz.DefaultOfBizListIterator$LookaheadIterator.<init>(DefaultOfBizListIterator.java:167)
at com.atlassian.jira.ofbiz.DefaultOfBizListIterator$LookaheadIterator.<init>(DefaultOfBizListIterator.java:161)
at com.atlassian.jira.ofbiz.DefaultOfBizListIterator.iterator(DefaultOfBizListIterator.java:153)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:230)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.consumeWith(SelectQueryImpl.java:214)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.singleValue(SelectQueryImpl.java:191)
at com.atlassian.jira.scheduler.OfBizClusteredJobDao.find(OfBizClusteredJobDao.java:88)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.getJobDetails(CaesiumSchedulerService.java:230)
at com.atlassian.scheduler.core.DelegatingSchedulerService.getJobDetails(DelegatingSchedulerService.java:97)
at com.atlassian.jira.service.DefaultServiceManager.ensureServiceScheduled(DefaultServiceManager.java:530)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.atlassian.jira.service.DefaultServiceManager.ensureServicesScheduled(DefaultServiceManager.java:523)
at com.atlassian.jira.service.DefaultServiceManager.start(DefaultServiceManager.java:104)
at com.atlassian.jira.component.pico.ComponentManager.runStartable(ComponentManager.java:325)
... 12 more
Caused by: org.ofbiz.core.entity.GenericDataSourceException: SQL Exception while getting value: (IO Exception: "java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]" [90028-185])
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.getValue(SqlJdbcUtil.java:700)
at org.ofbiz.core.entity.EntityListIterator.currentGenericValue(EntityListIterator.java:181)
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:267)
... 37 more
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception: "java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]" [90028-185]
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:287)
at org.h2.message.DbException.toSQLException(DbException.java:268)
at org.h2.message.TraceObject.logAndConvert(TraceObject.java:352)
at org.h2.jdbc.JdbcBlob.getBytes(JdbcBlob.java:101)
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.getBlobAsByteArray(SqlJdbcUtil.java:747)
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.getValue(SqlJdbcUtil.java:695)
... 39 more
Caused by: java.io.IOException: Block not found in id [1, -15, 2, -119, 20] [1.4.185/50]
at org.h2.mvstore.StreamStore$Stream.read(StreamStore.java:427)
at org.h2.util.IOUtils.copy(IOUtils.java:167)
at org.h2.jdbc.JdbcBlob.getBytes(JdbcBlob.java:95)
... 41 more
Caused by: java.lang.IllegalStateException: Reading from cache:nio:C:/Program Files/Atlassian/Application Data/JIRA/database/h2db.mv.db failed; file length 205557760 read length 768 at 207003968 [1.4.185/1]
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:443)
at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
at org.h2.mvstore.Page.read(Page.java:189)
at org.h2.mvstore.MVStore.readPage(MVStore.java:1830)
at org.h2.mvstore.MVMap.readPage(MVMap.java:736)
at org.h2.mvstore.Page.getChildPage(Page.java:216)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:468)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:469)
at org.h2.mvstore.MVMap.get(MVMap.java:450)
at org.h2.mvstore.StreamStore.getBlock(StreamStore.java:347)
at org.h2.mvstore.StreamStore$Stream.nextBuffer(StreamStore.java:464)
at org.h2.mvstore.StreamStore$Stream.read(StreamStore.java:421)
... 43 more
Caused by: java.io.EOFException
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:431)
... 54 more
2019-03-04 08:57:01,530 JIRA-Bootstrap ERROR [c.a.jira.health.HealthChecks] We couldn't start JIRA
2019-03-04 08:57:01,530 JIRA-Bootstrap ERROR [c.a.jira.health.HealthChecks] An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.
See our documentation for more information on contacting our support team and creating a support zip.
2019-03-04 08:57:01,671 JIRA-Bootstrap ERROR [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: startup-unexpected) , Desc = We couldn't start JIRA , Exception = An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.<br/>See our documentation for more information on contacting our support team and creating a support zip.)]
2019-03-04 08:57:01,686 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 833 MiB. Committed: 3387 MiB. Max: 4009 MiB
Non-heap memory : Used: 326 MiB. Committed: 351 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 1159 MiB. Committed: 3738 MiB. Max: 5545 MiB
---------------------------------------------------------------------------------
2019-03-04 08:57:04,233 Timer-0 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.infosysta.jira.mobile]
2019-03-04 08:57:40,355 hipchat-plugin-tasks-executor-0 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.plugins.base-hipchat-integration-plugin-api]

 

 

1 answer

3 votes
Petr Vaníček
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 4, 2019

Hi,

are you using HSQL as DBMS? It looks like you have probably corrupted DB files. So I hope you are not using it in production (based on your JVM values), because HSQL is only for testing purposes. My recommendation is 

  1. Try restart application again
  2. If it is important for you try recover DB files from backups (if you have backups) and then migrate to normal DBMS.
  3. If it's not working then install some kind of DBMS (I recommend PostgreSQL) and configure Jira with it, but you will have clean Jira.

Let me know how is it going.

Stefano Sbalchiero March 4, 2019

uhm... no i have internal db, i know we need to configure external db but i did'nt have time to do this...

 

i have a vm backup.. there are any other way to recover bd data?

Petr Vaníček
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 4, 2019

HSQL is only for testing purposes so if you don't have any DB file backups then there is nothing.

So you can try recover it by VM backup only and ASAP install some DBMS and migrate it - it's job for few minutes so nothing hard and time consuming.

Stefano Sbalchiero March 4, 2019

OK,

how can i migrate to esternal db?

Petr Vaníček
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 4, 2019

You can follow this tutorial

https://confluence.atlassian.com/jirakb/jira-how-to-migrate-from-hsql-database-to-an-external-database-295308246.html

But if you are not familiar with those things then you should ask someone with more experience.

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
March 4, 2019

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events