BitBucket fails every 30 minutes Edited

Hi,

My BitBucket server fails every 30 mins. Over the weekend it was upgraded from Stash 3.0.1 to BitBucket 5.6.1 in 3 steps:

- upgrade from 3.0.1 to 3.11
- upgrade from 3.11 to 4.14
- upgrade from 4.14 to 5.6.1

My BitBucket instance is integrated with JIRA, the application was recreated. It can use from 512-4096MB RAM and is based on internal HSQL database. I also increated the pool size to 200 but it did not help. 

Please have a look at the below exception maybe you can suggest some potential reason of this issue.

Thank you for any help,
Rafi

2017-12-12 09:22:10,385 ERROR [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMMLKYC/repos/ammlkyc/pull-requests/2281/comments HTTP/1.1" o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms.
2017-12-12 09:22:10,387 DEBUG [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMLKYC/repos/amlkyc/pull-requests/2281/comments HTTP/1.1" c.a.s.i.r.e.ServiceExceptionMapper Mapping ServiceException to REST response 500
com.atlassian.bitbucket.DataStoreException: A database error has occurred.
at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:45)
at com.atlassian.stash.internal.permission.GrantedPermissionVoterProvider.lambda$loadGroupsByUser$2(GrantedPermissionVoterProvider.java:238)
..
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.lang.Thread.run(Thread.java:745)
... 297 frames trimmed
Caused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
..
at com.atlassian.stash.internal.crowd.RiotPolice.findGroupsByUser(RiotPolice.java:301)
at com.atlassian.stash.internal.user.DefaultUserService.findGroupsByUser(DefaultUserService.java:200)
... 64 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:603)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:193)
..
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
... 95 common frames omitted
2017-12-12 09:22:12,248 ERROR [http-nio-7990-exec-92] @182YHMSx561x6889x109 10.244.16.24 "GET /rest/remote-link-aggregation/latest/aggregation HTTP/1.1" c.a.p.r.c.e.j.ThrowableExceptionMapper Uncaught exception thrown by REST service: There was a SQL exception thrown by the Active Objects library:
null
null

java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
null
null

java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:142)
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.find(TenantAwareActiveObjects.java:296)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

 

2 answers

2 accepted

You should not be using the HSQL database for production use.  It's meant for testing only.  It's pretty easy to migrate to an external database and documented by Atlassian. I would recommend using Postgres, as it is supported by Server and Data Center.

Thank you.

We found information in the logs that the pool size of HSQL database is consumed entirely:

2017-12-13 03:16:39,223 WARN  [Caesium-1-3]  c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.

Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values

- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL

We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.

Thank you,
Rafal

We found information in the logs that the pool size of HSQL database is consumed entirely:

2017-12-13 03:16:39,223 WARN  [Caesium-1-3]  c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.

Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values

- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL

We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.

Thank you,
Rafal

There are ways to increase the pool size, but they're not a lot of use because they cause other problems.

You really need to move off HSQL.

We migrate our database from HSQL to Oracle 12c and the problem was resolved by that. Thank you for help.

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Posted Jun 12, 2018 in Bitbucket

Do you use any Atlassian products for your personal projects?

After spinning my wheels trying to get organized enough to write a book for National Novel Writing Month (NaNoWriMo) I took my affinity for Atlassian products from my work life and decided to tr...

26,290 views 26 12
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you