Agent shutdown with message : "Agent will attempt to recover its normal operation"

[Maximilian Friedmann]

Good morning,

several agents are having these problems in our system. The agent kills itself during a job execution with the following message :

{code}

org.springframework.transaction.UnexpectedRollbackException: JTA transaction unexpectedly rolled back (maybe due to a timeout); nested exception is javax.transaction.RollbackException: Error during one-phase commit
Caused by: javax.transaction.RollbackException: Error during one-phase commit
	at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:311)
	at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
	at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:842)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
	at org.jencks.GeronimoPlatformTransactionManager.commit(GeronimoPlatformTransactionManager.java:76)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy0.takeBuildContext(Unknown Source)
	at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.getBuildContextFromQueue(BuildAgentControllerImpl.java:180)
	at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:73)
	at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:106)
	at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$2.run(NamedThreadFactory.java:50)
	at java.lang.Thread.run(Thread.java:662)
Caused by: javax.transaction.xa.XAException: Transaction 'XID:1197822575:010000000000000036c40e00c0a89261000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000:010000000000000036c40e00c0a89261000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' has not been started.
	at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:729)
	at org.apache.activemq.TransactionContext.commit(TransactionContext.java:549)
	at org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:85)
	at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.commit(WrapperNamedXAResource.java:47)
	at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:302)
	... 15 more
Caused by: javax.transaction.xa.XAException: Transaction 'XID:1197822575:010000000000000036c40e00c0a89261000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000:010000000000000036c40e00c0a89261000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' has not been started.
	at org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:290)
	at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:172)
	at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
	at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:424)
	at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
	at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:228)
	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
	at java.lang.Thread.run(Unknown Source)

{code}

Can anybody help?

[Przemek Bruski · Accepted solution]

Support now has a fix that will work with any Bamboo version. When contacting support, please give BSP-4822 as reference. The latest Bamboo version, 3.2 is already fixed.

[Przemek Bruski]

Support now has a fix that will work with any Bamboo version. Please give BSP-4822 as reference. The latest Bamboo version, 3.2 is already fixed.

[Qingsong Zhang]

We have the same issue with 3.3.2.

[Przemek Bruski]

Please open an issue at support.atlassian.com. It's unlikely that you're affected by exactly the same issue, it's been fixed in Bamboo 3.2.

[Qingsong Zhang]

any update

[Maximilian Friedmann]

It seems to only occur on hosts where more than one agent is running. The most it happens on a host where 8 Agents are running at the same time, does that help?

[Przemek Bruski]

Yup, that's exactly when it happens for other clients. 3.1.4 is also affected, we will either issue a binary patch or give a workaround for this problem.

[Maximilian Friedmann]

Hi,

unfortunately I cannot find an issue called BSP-4822, can you post a link?

[Przemek Bruski]

Issues raised by our clients are private. Please open a new one and give BSP-4822 as a reference.

[Maximilian Friedmann]

We just updated to version 3.1.4 and the problem still occurs...