Jira Server Keeps Crashing

I think it might be because we have a LOT of tickets. Is there a max amount of tickets?

I read other questions about running out of memory, but they don't really say if it causes the crash or how to solve it. 

Jira will come back up once I restart Jira and Apache

1 answer

0 vote

We've seen Jira instances with millions of issues, but they're mostly all now on Data Centre installs.

The reason "they don't really say if it causes the crash or how to solve it" is because there are many reasons it might be crashing, and there's no way to say that it is "too many issues" without looking at it.

What you need to do is look through the logs to find out why Jira is crashing.  Start with <jira home>/logs/atlassian-jira.log and <jira install>/log/catalina.out

So the error I'm seeing is

2017-10-08 05:20:03,299 http-nio-8080-exec-979 ERROR anonymous 319x260738x52 - 10.128.0.111 /rest/api/2/issue [NoModule] There was an error getting a DBCP datasource.

java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:52)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:80)

        at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)

        at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:173)

        at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:114)

        at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:340)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:456)

        at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:870)

        at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:857)

        at org.ofbiz.core.entity.GenericHelperDAO.findListIteratorByCondition(GenericHelperDAO.java:216)

        at org.ofbiz.core.entity.GenericDelegator.findListIteratorByCondition(GenericDelegator.java:1243)

        at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:398)

        at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findListIteratorByCondition(WrappingOfBizDelegator.java:278)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:227)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.consumeWith(SelectQueryImpl.java:214)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.singleValue(SelectQueryImpl.java:191)

        at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getCredential(OfBizUserDao.java:265)

        at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.getCredential(DelegatingUserDao.java:83)

        at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.getCredential(IndexedUserDao.java:63)

        at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.getCredential(DelegatingUserDao.java:83)

        at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.getCredential(SwitchingUserDao.java:30)

        at com.atlassian.crowd.directory.AbstractInternalDirectory.processAuthentication(AbstractInternalDirectory.java:254)

        at com.atlassian.crowd.directory.AbstractInternalDirectory.authenticate(AbstractInternalDirectory.java:222)

        at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:311)

        at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:198)

        at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:75)

        at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)

        at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)

        at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:88)

        ... 13 filtered

        at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:37)

        ... 19 filtered

        at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)

        ... 10 filtered

        at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)

        ... 4 filtered

        at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)

        ... 26 filtered

        at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)

        ... 23 filtered

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:285)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:48)

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449)

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)

        ... 138 more

2017-10-08 05:20:11,260 Caesium-1-1 ERROR ServiceRunner     [NoModule] There was an error getting a DBCP datasource.

java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:52)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:80)

        at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)

        at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:173)

        at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:114)

        at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:340)

        at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:456)

        at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(GenericDAO.java:870)

        at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:857)

        at org.ofbiz.core.entity.GenericHelperDAO.findListIteratorByCondition(GenericHelperDAO.java:216)

        at org.ofbiz.core.entity.GenericDelegator.findListIteratorByCondition(GenericDelegator.java:1243)

        at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:398)

        at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findListIteratorByCondition(WrappingOfBizDelegator.java:278)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:227)

        at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.asList(SelectQueryImpl.java:164)

        at com.atlassian.jira.scheduler.OfBizRunDetailsDao.getAllIdsForJob(OfBizRunDetailsDao.java:139)

        at com.atlassian.jira.scheduler.OfBizRunDetailsDao.addRunDetails(OfBizRunDetailsDao.java:111)

        at com.atlassian.scheduler.core.AbstractSchedulerService.addRunDetails(AbstractSchedulerService.java:151)

        at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:98)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:410)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:388)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)

        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)

        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)

        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)

        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:285)

        at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:283)

        at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:48)

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449)

        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)

        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)

        ... 33 more

Here is the allocated memory, so it doesnt seem like the problem, I think

      Heap memory     :  Used:  298 MiB.  Committed:  712 MiB.  Max:  737 MiB

      Non-heap memory :  Used:  232 MiB.  Committed:  253 MiB.  Max: 1264 MiB

    ---------------------------------------------------------------------------------

      TOTAL           :  Used:  530 MiB.  Committed:  965 MiB.  Max: 2001 MiB

That looks like a startup error where it can't even connect to the database.  Is it starting at all?  Or is this when a crash happens?

Yeah, it will start up after we reset it. The error message is what prints when it crashes

Ok, that tells us it is losing the connection to the database.  Or rather, it's trying to get a connection and it can't.

Is there any pattern to the crashing?   Like it's happening at 3am every day (i.e. the backups are kicking in and taking up all the database connections)

I guess it crashes at night. (but is really slow during the day after we restart it). How would we solve the backup filling the DB connections? 

The backup is a guess, but it's happened to me more than once. 

The idea is that the database has a "pool" of connections so nothing gets overloaded.  If, for example, you've got a pool of 20, then applications can connect up to 20 times, with each connection running a query or batch of them, then quitting.  If it reaches the limit running at the same time, then any new connections are forced to wait until one of the entries in the pool is freed up.  In your case, it's having to wait too long for one to become clear, which suggests something else is consuming the connections.  Backups often grab connections and hold them for a very long time.

You'll need to talk to your DBAs about fixing this, either by limiting the backups so they don't interfere, or if it's ok to increase the limit for Jira.

Thank you! We gave Jira more memory (which definitely helped because now the GUI is actually loading, and telling us it cant reach the DB). 

We tried to update the amount of pools to 50, but it still couldnt connect. The other 2 suggestions given was the DB wasnt on or the username and password is missing. How do we tell if its on? And I dont remember setting a password, is there a default? 

We were also looking at the dbconfig.xml file and it references h2db directory but we just have h2db.mv.db and h2db.trace.db. Do we need to edit our XML file? 

This is the new error I'm getting:

2017-10-10 13:54:43,398 JIRA-Bootstrap INFO      [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml

2017-10-10 13:54:43,813 JIRA-Bootstrap ERROR      [c.a.config.bootstrap.DefaultAtlassianBootstrapManager] Could not successfully test your database:

org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-185]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)

at org.h2.message.DbException.get(DbException.java:168)

at org.h2.message.DbException.convert(DbException.java:295)

at org.h2.engine.Database.openDatabase(Database.java:297)

at org.h2.engine.Database.<init>(Database.java:260)

at org.h2.engine.Engine.openSession(Engine.java:60)

at org.h2.engine.Engine.openSession(Engine.java:167)

at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)

at org.h2.engine.Engine.createSession(Engine.java:128)

at org.h2.engine.Engine.createSession(Engine.java:26)

at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)

at org.h2.Driver.connect(Driver.java:72)

at java.sql.DriverManager.getConnection(DriverManager.java:664)

at java.sql.DriverManager.getConnection(DriverManager.java:247)

at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.getTestDatabaseConnection(DefaultAtlassianBootstrapManager.java:347)

at com.atlassian.jira.config.database.JdbcDatasource.getConnection(JdbcDatasource.java:211)

at com.atlassian.jira.config.database.DatabaseConfig.testConnection(DatabaseConfig.java:89)

at com.atlassian.jira.health.checks.DbConfigurationAndConnectionCheck.onDatabaseConfigured(DbConfigurationAndConnectionCheck.java:53)

at com.atlassian.jira.health.HealthCheckTemplate.apply(HealthCheckTemplate.java:29)

at com.atlassian.jira.health.HealthCheckTemplate.apply(HealthCheckTemplate.java:14)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.runCheck(DefaultHealthCheckExecutor.java:70)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.lambda$applyAndCollectExceptions$0(DefaultHealthCheckExecutor.java:51)

at java.lang.Iterable.forEach(Iterable.java:75)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.applyAndCollectExceptions(DefaultHealthCheckExecutor.java:51)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.apply(DefaultHealthCheckExecutor.java:42)

at com.atlassian.jira.health.DefaultHealthCheckExecutor.apply(DefaultHealthCheckExecutor.java:33)

at com.atlassian.jira.health.HealthChecks.executeChecksAndRecordResults(HealthChecks.java:165)

at com.atlassian.jira.health.HealthChecks.runHealthChecks(HealthChecks.java:155)

at com.atlassian.jira.health.HealthChecks.runHealthChecks(HealthChecks.java:67)

at com.atlassian.jira.startup.BootstrapContainerLauncher.start(BootstrapContainerLauncher.java:44)

at com.atlassian.jira.startup.DefaultJiraLauncher.preDbLaunch(DefaultJiraLauncher.java:118)

at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:104)

at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)

at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:103)

at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:157)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.NullPointerException

at org.h2.mvstore.db.ValueDataType.compare(ValueDataType.java:102)

at org.h2.mvstore.MVMap.compare(MVMap.java:708)

at org.h2.mvstore.Page.binarySearch(Page.java:333)

at org.h2.mvstore.MVMap.binarySearch(MVMap.java:461)

at org.h2.mvstore.MVMap.get(MVMap.java:450)

at org.h2.mvstore.db.TransactionStore$Transaction.commit(TransactionStore.java:762)

at org.h2.mvstore.db.MVTableEngine$Store.initTransactions(MVTableEngine.java:224)

at org.h2.engine.Database.open(Database.java:736)

at org.h2.engine.Database.openDatabase(Database.java:266)

... 34 more

2017-10-10 13:54:43,833 JIRA-Bootstrap ERROR      [c.a.jira.health.HealthChecks] JIRA couldn't connect to your database

2017-10-10 13:54:43,833 JIRA-Bootstrap ERROR      [c.a.jira.health.HealthChecks] JIRA failed to establish a connection to your database.

    This could be because:

    - Your database isn't running

    - The configuration of your dbconfig.xml file is incorrect (user, password, or database URL etc.)

    - There is a network issue between JIRA and your database (e.g. firewall, database doesn't allow remote access etc.)

    

    There are several other solutions you can try, review our documentation and see what works for you.

Suggest an answer

Log in or Join to answer
Community showcase
Sarah Schuster
Posted Jan 29, 2018 in Jira

What are common themes you've seen across successful & failed Jira Software implementations?

Hey everyone! My name is Sarah Schuster, and I'm a Customer Success Manager in Atlassian specializing in Jira Software Cloud. Over the next few weeks I will be posting discussion topics (8 total) to ...

3,323 views 14 20
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you
Atlassian Team Tour

Join us on the Team Tour

We're bringing product updates and pro tips on teamwork to ten cities around the world.

Save your spot