BitBucket fails every 30 minutes

Rafal Sojka December 12, 2017

Hi,

My BitBucket server fails every 30 mins. Over the weekend it was upgraded from Stash 3.0.1 to BitBucket 5.6.1 in 3 steps:

- upgrade from 3.0.1 to 3.11
- upgrade from 3.11 to 4.14
- upgrade from 4.14 to 5.6.1

My BitBucket instance is integrated with JIRA, the application was recreated. It can use from 512-4096MB RAM and is based on internal HSQL database. I also increated the pool size to 200 but it did not help. 

Please have a look at the below exception maybe you can suggest some potential reason of this issue.

Thank you for any help,
Rafi

2017-12-12 09:22:10,385 ERROR [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMMLKYC/repos/ammlkyc/pull-requests/2281/comments HTTP/1.1" o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms.
2017-12-12 09:22:10,387 DEBUG [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMLKYC/repos/amlkyc/pull-requests/2281/comments HTTP/1.1" c.a.s.i.r.e.ServiceExceptionMapper Mapping ServiceException to REST response 500
com.atlassian.bitbucket.DataStoreException: A database error has occurred.
at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:45)
at com.atlassian.stash.internal.permission.GrantedPermissionVoterProvider.lambda$loadGroupsByUser$2(GrantedPermissionVoterProvider.java:238)
..
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.lang.Thread.run(Thread.java:745)
... 297 frames trimmed
Caused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
..
at com.atlassian.stash.internal.crowd.RiotPolice.findGroupsByUser(RiotPolice.java:301)
at com.atlassian.stash.internal.user.DefaultUserService.findGroupsByUser(DefaultUserService.java:200)
... 64 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:603)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:193)
..
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
... 95 common frames omitted
2017-12-12 09:22:12,248 ERROR [http-nio-7990-exec-92] @182YHMSx561x6889x109 10.244.16.24 "GET /rest/remote-link-aggregation/latest/aggregation HTTP/1.1" c.a.p.r.c.e.j.ThrowableExceptionMapper Uncaught exception thrown by REST service: There was a SQL exception thrown by the Active Objects library:
null
null

java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
null
null

java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:142)
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.find(TenantAwareActiveObjects.java:296)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

 

2 answers

2 accepted

0 votes
Answer accepted
Rafal Sojka December 13, 2017

We found information in the logs that the pool size of HSQL database is consumed entirely:

2017-12-13 03:16:39,223 WARN  [Caesium-1-3]  c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.

Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values

- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL

We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.

Thank you,
Rafal

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
December 13, 2017

There are ways to increase the pool size, but they're not a lot of use because they cause other problems.

You really need to move off HSQL.

Rafal Sojka December 13, 2017

Ok, thank you.

Rafal Sojka January 5, 2018

We migrate our database from HSQL to Oracle 12c and the problem was resolved by that. Thank you for help.

0 votes
Answer accepted
Dave Theodore [Coyote Creek Consulting]
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
December 12, 2017

You should not be using the HSQL database for production use.  It's meant for testing only.  It's pretty easy to migrate to an external database and documented by Atlassian. I would recommend using Postgres, as it is supported by Server and Data Center.

Rafal Sojka December 13, 2017

Thank you.

We found information in the logs that the pool size of HSQL database is consumed entirely:

2017-12-13 03:16:39,223 WARN  [Caesium-1-3]  c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.

Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values

- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL

We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.

Thank you,
Rafal

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events