What is a reason of "504 Gateway Time-out"

December 28, 2022

hello,

Today I had problem with Bitbucket "504 Gateway Time-out" - after container restart it works but can you verify logs what could cause this problem and how I can avoid it in future?

Details:

Version: Atlassian Bitbucket v5.0.10

Env: docker container

DB: postgresql

What I noticed?

OOM problem:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-69-thread-1-Logging Task Scheduler"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "io-pump:thread-116302"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 19"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 25"

DB problem:

2022-12-28 13:33:46,038 ERROR [http-nio-7990-exec-1465] ... "GET /projects/ HTTP/1.1" o.a.c.c.C.[.[.[/].[springMvc] Servlet.service() for servlet springMvc threw exception

com.atlassian.bitbucket.DataStoreException: A database error has occurred.

at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:45)

at com.atlassian.stash.internal.spring.security.HttpSessionSecurityContextRepository.loadContext(HttpSessionSecurityContextRepository.java:71)

at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)

at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)

at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)

at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)

at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)

at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)

at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)

at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:32)

at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:38)

at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)

at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)

at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)

at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)

at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:89)

at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.lang.Thread.run(Thread.java:748)

... 169 frames trimmed

aused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement

at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)

at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)

at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148)

at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1940)

at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1909)

at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887)

at org.hibernate.loader.Loader.doQuery(Loader.java:932)

at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)

at org.hibernate.loader.Loader.doList(Loader.java:2615)

at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2460)

at org.hibernate.loader.Loader.list(Loader.java:2422)

at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:109)

at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1896)

at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:366)

at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:388)

at com.atlassian.stash.internal.crowd.HibernateUserDao.internalFindByName(HibernateUserDao.java:449)

at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:128)

at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:69)

at com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(AbstractInternalDirectory.java:179)

at com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(AbstractInternalDirectory.java:66)

at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.findUserByName(DirectoryManagerGeneric.java:323)

at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.findUserByName(ApplicationServiceGeneric.java:368)

at com.atlassian.crowd.embedded.core.CrowdServiceImpl.getUser(CrowdServiceImpl.java:100)

at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.getCrowdUser(CrowdAwareLoadListener.java:130)

at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.setCrowdUserIfNecessary(CrowdAwareLoadListener.java:149)

at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.onPostLoad(CrowdAwareLoadListener.java:87)

at org.hibernate.event.internal.DefaultLoadEventListener.convertCacheEntryToEntity(DefaultLoadEventListener.java:801)

at org.hibernate.event.internal.DefaultLoadEventListener.processCachedEntry(DefaultLoadEventListener.java:625)

at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:602)

at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:462)

at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219)

at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:278)

at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:121)

at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89)

at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.onLoad(CrowdAwareLoadListener.java:69)

at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1237)

at org.hibernate.internal.SessionImpl.access$1900(SessionImpl.java:203)

at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.doLoad(SessionImpl.java:2798)

at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2772)

at org.hibernate.internal.SessionImpl.get(SessionImpl.java:1073)

at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:89)

at com.atlassian.stash.internal.AbstractHibernateDao.getById(AbstractHibernateDao.java:28)

at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:295)

at com.atlassian.stash.internal.user.DefaultUserService.getUserById(DefaultUserService.java:289)

... 20 common frames omitted

Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15914ms.

at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)

at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)

at com.atlassian.stash.internal.hikari.ExtendedHikariDataSource.getConnection(ExtendedHikariDataSource.java:127)

at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)

at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)

... 62 common frames omitted

Next exception: org.postgresql.util.PSQLException: Connection attempt timed out.

at org.postgresql.Driver$ConnectThread.getResult(Driver.java:371)

at org.postgresql.Driver.connect(Driver.java:282)

at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)

at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)

at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)

at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)

at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)

at com.zaxxer.hikari.pool.HikariPool.access$200(HikariPool.java:71)

at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:631)

at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:617)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.