It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

BitBucket fails every 30 minutes Edited

Hi,

My BitBucket server fails every 30 mins. Over the weekend it was upgraded from Stash 3.0.1 to BitBucket 5.6.1 in 3 steps:

- upgrade from 3.0.1 to 3.11
- upgrade from 3.11 to 4.14
- upgrade from 4.14 to 5.6.1

My BitBucket instance is integrated with JIRA, the application was recreated. It can use from 512-4096MB RAM and is based on internal HSQL database. I also increated the pool size to 200 but it did not help. 

Please have a look at the below exception maybe you can suggest some potential reason of this issue.

Thank you for any help,
Rafi

2017-12-12 09:22:10,385 ERROR [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMMLKYC/repos/ammlkyc/pull-requests/2281/comments HTTP/1.1" o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms.
2017-12-12 09:22:10,387 DEBUG [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMLKYC/repos/amlkyc/pull-requests/2281/comments HTTP/1.1" c.a.s.i.r.e.ServiceExceptionMapper Mapping ServiceException to REST response 500
com.atlassian.bitbucket.DataStoreException: A database error has occurred.
at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:45)
at com.atlassian.stash.internal.permission.GrantedPermissionVoterProvider.lambda$loadGroupsByUser$2(GrantedPermissionVoterProvider.java:238)
..
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.lang.Thread.run(Thread.java:745)
... 297 frames trimmed
Caused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
..
at com.atlassian.stash.internal.crowd.RiotPolice.findGroupsByUser(RiotPolice.java:301)
at com.atlassian.stash.internal.user.DefaultUserService.findGroupsByUser(DefaultUserService.java:200)
... 64 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:603)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:193)
..
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
... 95 common frames omitted
2017-12-12 09:22:12,248 ERROR [http-nio-7990-exec-92] @182YHMSx561x6889x109 10.244.16.24 "GET /rest/remote-link-aggregation/latest/aggregation HTTP/1.1" c.a.p.r.c.e.j.ThrowableExceptionMapper Uncaught exception thrown by REST service: There was a SQL exception thrown by the Active Objects library:
null
null

java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
null
null

java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:142)
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.find(TenantAwareActiveObjects.java:296)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

 

2 answers

2 accepted

0 votes
Answer accepted

You should not be using the HSQL database for production use.  It's meant for testing only.  It's pretty easy to migrate to an external database and documented by Atlassian. I would recommend using Postgres, as it is supported by Server and Data Center.

Thank you.

We found information in the logs that the pool size of HSQL database is consumed entirely:

2017-12-13 03:16:39,223 WARN  [Caesium-1-3]  c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.

Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values

- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL

We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.

Thank you,
Rafal

0 votes
Answer accepted

We found information in the logs that the pool size of HSQL database is consumed entirely:

2017-12-13 03:16:39,223 WARN  [Caesium-1-3]  c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.

Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values

- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL

We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.

Thank you,
Rafal

Nic Brough Community Leader Dec 13, 2017

There are ways to increase the pool size, but they're not a lot of use because they cause other problems.

You really need to move off HSQL.

Ok, thank you.

We migrate our database from HSQL to Oracle 12c and the problem was resolved by that. Thank you for help.

Suggest an answer

Log in or Sign up to answer
Community showcase
Published in Bitbucket

Atlassian supported Jenkins integration for Bitbucket Server

We’ve been building a plugin to integrate Bitbucket Server and Jenkins CI, and I’m excited to announce that our alpha is ready to download and install. It lets you seamlessly configure a Jenkins job ...

1,091 views 3 17
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you