Hi,
My BitBucket server fails every 30 mins. Over the weekend it was upgraded from Stash 3.0.1 to BitBucket 5.6.1 in 3 steps:
- upgrade from 3.0.1 to 3.11
- upgrade from 3.11 to 4.14
- upgrade from 4.14 to 5.6.1
My BitBucket instance is integrated with JIRA, the application was recreated. It can use from 512-4096MB RAM and is based on internal HSQL database. I also increated the pool size to 200 but it did not help.
Please have a look at the below exception maybe you can suggest some potential reason of this issue.
Thank you for any help,
Rafi
2017-12-12 09:22:10,385 ERROR [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMMLKYC/repos/ammlkyc/pull-requests/2281/comments HTTP/1.1" o.h.e.jdbc.spi.SqlExceptionHelper bitbucket - Connection is not available, request timed out after 15000ms.
2017-12-12 09:22:10,387 DEBUG [http-nio-7990-exec-80] me @182YHMSx561x6902x112 g633tk 10.180.102.130 "POST /rest/api/latest/projects/AMLKYC/repos/amlkyc/pull-requests/2281/comments HTTP/1.1" c.a.s.i.r.e.ServiceExceptionMapper Mapping ServiceException to REST response 500
com.atlassian.bitbucket.DataStoreException: A database error has occurred.
at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:45)
at com.atlassian.stash.internal.permission.GrantedPermissionVoterProvider.lambda$loadGroupsByUser$2(GrantedPermissionVoterProvider.java:238)
..
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.lang.Thread.run(Thread.java:745)
... 297 frames trimmed
Caused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
..
at com.atlassian.stash.internal.crowd.RiotPolice.findGroupsByUser(RiotPolice.java:301)
at com.atlassian.stash.internal.user.DefaultUserService.findGroupsByUser(DefaultUserService.java:200)
... 64 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:603)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:193)
..
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
... 95 common frames omitted
2017-12-12 09:22:12,248 ERROR [http-nio-7990-exec-92] @182YHMSx561x6889x109 10.244.16.24 "GET /rest/remote-link-aggregation/latest/aggregation HTTP/1.1" c.a.p.r.c.e.j.ThrowableExceptionMapper Uncaught exception thrown by REST service: There was a SQL exception thrown by the Active Objects library:
null
null
java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
null
null
java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:142)
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.find(TenantAwareActiveObjects.java:296)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
We found information in the logs that the pool size of HSQL database is consumed entirely:
2017-12-13 03:16:39,223 WARN [Caesium-1-3] c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.
Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values
- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL
We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.
Thank you,
Rafal
There are ways to increase the pool size, but they're not a lot of use because they cause other problems.
You really need to move off HSQL.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
We migrate our database from HSQL to Oracle 12c and the problem was resolved by that. Thank you for help.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You should not be using the HSQL database for production use. It's meant for testing only. It's pretty easy to migrate to an external database and documented by Atlassian. I would recommend using Postgres, as it is supported by Server and Data Center.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thank you.
We found information in the logs that the pool size of HSQL database is consumed entirely:
2017-12-13 03:16:39,223 WARN [Caesium-1-3] c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow.
Is there any way to increase the HSQL pool size ? We have tried various options:
- manual changes in db.properties - but this file was overwritten after each restart with original values
- db-related parameters in bitbucket.properties but it seems they are not valid for HSQL
We are waiting for proper Oracle database but in the meantime it would be great if there is any way to tune HSQL parameters to make it work longer than 30mins.
Thank you,
Rafal
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.