Bamboo stops triggering builds

[Aske Jenstrup Olsson]

Hi,
I recently upgraded our bamboo instance from v6.7.2 to 6.9.1 and enabled TLS through tomcat and our existing ssl cert. 6.7.2 was running plain http.

However since the upgradle bamboo has stopped triggering new builds (through polling and scheduled) after an hour or two. A restart will temporary bring the triggering back...
I can't find much in the logs, though I see the following stacktrace 2-4 times in the bamboo-stderr log

05-Jul-2019 09:35:14.611 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [https-openssl-nio-443-exec-11] (id=[237]) has been active for [69,412] milliseconds (since [7/5/19 9:34 AM]) to serve the same request for [https://bamboo.xxxx.com/userlogin!doDefault.action?os_destination=%2Fchain%2Fadmin%2Fajax%2FgetChains.action%3FplanKey%3DPD-BUILD%26_%3D1562229809241] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:282)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1898)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:117)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:379)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:60)

But I guess this only informs that some thread has been running for more than 60 sseonds.

A thread-dump only shows threads with the folloing states:

java.lang.Thread.State: RUNNABLE
java.lang.Thread.State: TIMED_WAITING (on object monitor)
java.lang.Thread.State: TIMED_WAITING (parking)
java.lang.Thread.State: TIMED_WAITING (sleeping)
java.lang.Thread.State: WAITING (on object monitor)
java.lang.Thread.State: WAITING (parking)

Any suggestions?

Regards
Aske

[Aske Jenstrup Olsson]

I also see this error in the "System Errors"

System Error Details

Build Crosscut - Build - xxxxx-127 : Plan 'CROS-CI1516-127' started but a fatal error occurred
(java.lang.IllegalArgumentException : Unknown RepositoryDefinition 148504626)
Occurred: 05 Jul 2019, 5:24:48 PM

---

java.lang.IllegalArgumentException: Unknown RepositoryDefinition 148504626
 at com.atlassian.bamboo.v2.build.BuildChangesImpl.getBuildRepositoryChanges(BuildChangesImpl.java:183)
 at com.atlassian.bamboo.v2.build.BuildChangesImpl.setPreviousVcsRevisionKey(BuildChangesImpl.java:75)
 at com.atlassian.bamboo.plan.vcsRevision.PlanVcsRevisionHistoryManagerImpl.markBuildStartedNewTx(PlanVcsRevisionHistoryManagerImpl.java:52)
 at sun.reflect.GeneratedMethodAccessor2062.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
 at com.sun.proxy.$Proxy246.markBuildStartedNewTx(Unknown Source)
 at com.atlassian.bamboo.plan.vcsRevision.PlanVcsRevisionHistoryServiceImpl.lambda$markBuildStarted$1(PlanVcsRevisionHistoryServiceImpl.java:31)
 at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:302)
 at com.atlassian.bamboo.plan.vcsRevision.PlanVcsRevisionHistoryServiceImpl.markBuildStarted(PlanVcsRevisionHistoryServiceImpl.java:30)
 at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.tryStartChainState(ChainExecutionManagerImpl.java:341)
 at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.delayedStart(ChainExecutionManagerImpl.java:241)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
 at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
 at com.sun.proxy.$Proxy230.delayedStart(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:206)
 at com.sun.proxy.$Proxy230.delayedStart(Unknown Source)
 at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1$1.lambda$call$0(PlanExecutionManagerImpl.java:375)
 at com.atlassian.bamboo.util.CacheAwareness.lambda$withValuesOlderThanTimestampReloaded$1(CacheAwareness.java:101)
 at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:126)
 at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:100)
 at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:151)
 at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1$1.call(PlanExecutionManagerImpl.java:374)
 at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1$1.call(PlanExecutionManagerImpl.java:367)
 at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:293)
 at com.atlassian.bamboo.plan.PlanExecutionLockServiceImpl.lock(PlanExecutionLockServiceImpl.java:75)
 at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.doWithProcessLock(PlanExecutionManagerImpl.java:652)
 at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.access$400(PlanExecutionManagerImpl.java:131)
 at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1.run(PlanExecutionManagerImpl.java:363)
 at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
 at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
 at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
 at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
 at java.lang.Thread.run(Thread.java:745)