Jira Server Keeps Crashing

[Holly Parrish]

I think it might be because we have a LOT of tickets. Is there a max amount of tickets?

I read other questions about running out of memory, but they don't really say if it causes the crash or how to solve it. 

Jira will come back up once I restart Jira and Apache

[Nic Brough -Adaptavist-]

We've seen Jira instances with millions of issues, but they're mostly all now on Data Centre installs.

The reason "they don't really say if it causes the crash or how to solve it" is because there are many reasons it might be crashing, and there's no way to say that it is "too many issues" without looking at it.

What you need to do is look through the logs to find out why Jira is crashing.  Start with <jira home>/logs/atlassian-jira.log and <jira install>/log/catalina.out

[Holly Parrish]

So the error I'm seeing is

2017-10-08 05:20:03,299 http-nio-8080-exec-979 ERROR anonymous 319x260738x52 - 10.128.0.111 /rest/api/2/issue [NoModule] There was an error getting a DBCP datasource.

java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:52)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:80)

        at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)

        at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:173)

        at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:114)

        at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:340)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:456)

        at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:870)

        at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:857)

        at org.ofbiz.core.entity.GenericHelperDAO.findListIteratorByCondition(GenericHelperDAO.java:216)

        at org.ofbiz.core.entity.GenericDelegator.findListIteratorByCondition(GenericDelegator.java:1243)

        at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:398)

        at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findListIteratorByCondition(WrappingOfBizDelegator.java:278)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:227)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.consumeWith(SelectQueryImpl.java:214)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.singleValue(SelectQueryImpl.java:191)

        at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getCredential(OfBizUserDao.java:265)

        at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.getCredential(DelegatingUserDao.java:83)

        at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.getCredential(IndexedUserDao.java:63)

        at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.getCredential(DelegatingUserDao.java:83)

        at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.getCredential(SwitchingUserDao.java:30)

        at com.atlassian.crowd.directory.AbstractInternalDirectory.processAuthentication(AbstractInternalDirectory.java:254)

        at com.atlassian.crowd.directory.AbstractInternalDirectory.authenticate(AbstractInternalDirectory.java:222)

        at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:311)

        at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:198)

        at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:75)

        at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)

        at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)

        at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:88)

        ... 13 filtered

        at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:37)

        ... 19 filtered

        at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)

        ... 10 filtered

        at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)

        ... 4 filtered

        at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)

        ... 26 filtered

        at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)

        ... 23 filtered

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:285)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:48)

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449)

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)

        ... 138 more

2017-10-08 05:20:11,260 Caesium-1-1 ERROR ServiceRunner     [NoModule] There was an error getting a DBCP datasource.

java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:52)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:80)

        at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)

        at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:173)

        at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:114)

        at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:340)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:456)

        at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:870)

        at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:857)

        at org.ofbiz.core.entity.GenericHelperDAO.findListIteratorByCondition(GenericHelperDAO.java:216)

        at org.ofbiz.core.entity.GenericDelegator.findListIteratorByCondition(GenericDelegator.java:1243)

        at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:398)

        at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findListIteratorByCondition(WrappingOfBizDelegator.java:278)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:227)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.asList(SelectQueryImpl.java:164)

        at com.atlassian.jira.scheduler.OfBizRunDetailsDao.getAllIdsForJob(OfBizRunDetailsDao.java:139)

        at com.atlassian.jira.scheduler.OfBizRunDetailsDao.addRunDetails(OfBizRunDetailsDao.java:111)

        at com.atlassian.scheduler.core.AbstractSchedulerService.addRunDetails(AbstractSchedulerService.java:151)

        at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:98)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:410)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:388)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)

        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)

        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)

        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:285)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:48)

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449)

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)

        ... 33 more

[Holly Parrish]

Here is the allocated memory, so it doesnt seem like the problem, I think

      Heap memory     :  Used:  298 MiB.  Committed:  712 MiB.  Max:  737 MiB

      Non-heap memory :  Used:  232 MiB.  Committed:  253 MiB.  Max: 1264 MiB

    ---------------------------------------------------------------------------------

      TOTAL           :  Used:  530 MiB.  Committed:  965 MiB.  Max: 2001 MiB

[Nic Brough -Adaptavist-]

That looks like a startup error where it can't even connect to the database.  Is it starting at all?  Or is this when a crash happens?

[Holly Parrish]

Yeah, it will start up after we reset it. The error message is what prints when it crashes

[Nic Brough -Adaptavist-]

Ok, that tells us it is losing the connection to the database.  Or rather, it's trying to get a connection and it can't.

Is there any pattern to the crashing?   Like it's happening at 3am every day (i.e. the backups are kicking in and taking up all the database connections)

[Holly Parrish]

I guess it crashes at night. (but is really slow during the day after we restart it). How would we solve the backup filling the DB connections? 

[Nic Brough -Adaptavist-]

The backup is a guess, but it's happened to me more than once. 

The idea is that the database has a "pool" of connections so nothing gets overloaded.  If, for example, you've got a pool of 20, then applications can connect up to 20 times, with each connection running a query or batch of them, then quitting.  If it reaches the limit running at the same time, then any new connections are forced to wait until one of the entries in the pool is freed up.  In your case, it's having to wait too long for one to become clear, which suggests something else is consuming the connections.  Backups often grab connections and hold them for a very long time.

You'll need to talk to your DBAs about fixing this, either by limiting the backups so they don't interfere, or if it's ok to increase the limit for Jira.

[Nic Brough -Adaptavist-]

Sorry, I missed out...

Have a read through

https://confluence.atlassian.com/jirakb/jira-stops-functioning-due-to-timeout-waiting-for-idle-object-718834030.html

and

https://confluence.atlassian.com/adminjiraserver070/surviving-connection-closures-749382657.html

before talking to the DBA

[Holly Parrish]

Thank you! We gave Jira more memory (which definitely helped because now the GUI is actually loading, and telling us it cant reach the DB). 

We tried to update the amount of pools to 50, but it still couldnt connect. The other 2 suggestions given was the DB wasnt on or the username and password is missing. How do we tell if its on? And I dont remember setting a password, is there a default? 

We were also looking at the dbconfig.xml file and it references h2db directory but we just have h2db.mv.db and h2db.trace.db. Do we need to edit our XML file? 

[Holly Parrish]

This is the new error I'm getting:

2017-10-10 13:54:43,398 JIRA-Bootstrap INFO      [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml

2017-10-10 13:54:43,813 JIRA-Bootstrap ERROR      [c.a.config.bootstrap.DefaultAtlassianBootstrapManager] Could not successfully test your database:

org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-185]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)

at org.h2.message.DbException.get(DbException.java:168)

at org.h2.message.DbException.convert(DbException.java:295)

at org.h2.engine.Database.openDatabase(Database.java:297)

at org.h2.engine.Database.<init>(Database.java:260)

at org.h2.engine.Engine.openSession(Engine.java:60)

at org.h2.engine.Engine.openSession(Engine.java:167)

at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)

at org.h2.engine.Engine.createSession(Engine.java:128)

at org.h2.engine.Engine.createSession(Engine.java:26)

at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)

at org.h2.Driver.connect(Driver.java:72)

at java.sql.DriverManager.getConnection(DriverManager.java:664)

at java.sql.DriverManager.getConnection(DriverManager.java:247)

at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.getTestDatabaseConnection(DefaultAtlassianBootstrapManager.java:347)

at com.atlassian.jira.config.database.JdbcDatasource.getConnection(JdbcDatasource.java:211)

at com.atlassian.jira.config.database.DatabaseConfig.testConnection(DatabaseConfig.java:89)

at com.atlassian.jira.health.checks.DbConfigurationAndConnectionCheck.onDatabaseConfigured(DbConfigurationAndConnectionCheck.java:53)

at com.atlassian.jira.health.HealthCheckTemplate.apply(HealthCheckTemplate.java:29)

at com.atlassian.jira.health.HealthCheckTemplate.apply(HealthCheckTemplate.java:14)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.runCheck(DefaultHealthCheckExecutor.java:70)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.lambda$applyAndCollectExceptions$0(DefaultHealthCheckExecutor.java:51)

at java.lang.Iterable.forEach(Iterable.java:75)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.applyAndCollectExceptions(DefaultHealthCheckExecutor.java:51)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.apply(DefaultHealthCheckExecutor.java:42)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.apply(DefaultHealthCheckExecutor.java:33)

at com.atlassian.jira.health.HealthChecks.executeChecksAndRecordResults(HealthChecks.java:165)

at com.atlassian.jira.health.HealthChecks.runHealthChecks(HealthChecks.java:155)

at com.atlassian.jira.health.HealthChecks.runHealthChecks(HealthChecks.java:67)

at com.atlassian.jira.startup.BootstrapContainerLauncher.start(BootstrapContainerLauncher.java:44)

at com.atlassian.jira.startup.DefaultJiraLauncher.preDbLaunch(DefaultJiraLauncher.java:118)

at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:104)

at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)

at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:103)

at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:157)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.NullPointerException

at org.h2.mvstore.db.ValueDataType.compare(ValueDataType.java:102)

at org.h2.mvstore.MVMap.compare(MVMap.java:708)

at org.h2.mvstore.Page.binarySearch(Page.java:333)

at org.h2.mvstore.MVMap.binarySearch(MVMap.java:461)

at org.h2.mvstore.MVMap.get(MVMap.java:450)

at org.h2.mvstore.db.TransactionStore$Transaction.commit(TransactionStore.java:762)

at org.h2.mvstore.db.MVTableEngine$Store.initTransactions(MVTableEngine.java:224)

at org.h2.engine.Database.open(Database.java:736)

at org.h2.engine.Database.openDatabase(Database.java:266)

... 34 more

2017-10-10 13:54:43,833 JIRA-Bootstrap ERROR      [c.a.jira.health.HealthChecks] JIRA couldn't connect to your database

2017-10-10 13:54:43,833 JIRA-Bootstrap ERROR      [c.a.jira.health.HealthChecks] JIRA failed to establish a connection to your database.

    This could be because:

    - Your database isn't running

    - The configuration of your dbconfig.xml file is incorrect (user, password, or database URL etc.)

    - There is a network issue between JIRA and your database (e.g. firewall, database doesn't allow remote access etc.)

    There are several other solutions you can try, review our documentation and see what works for you.