Build had to be cancelled: it was marked as building on XYZ but that agent is offline
Morning Team,
We are having problems with some builds in our Bamboo service (Server version 9.1.0 build 90100)
We are running all our builds using elastic bamboo EC2 instances with a custom image based on this specific bamboo ami (ami-0c8f07c41484b4f08
|| bamboo-elastic-ami-ubuntu-1674213244)
|| bamboo-elastic-ami-ubuntu-1674213244)
Some builds stopped after running for more than 20 minutes and the message received is (See screenshot || name: BambooAgentOffline.png):
Build AHP-DCBC8-C80IT-26 had to be cancelled: it was marked as building on 317358089 but that agent is offline
Having a look at the bamboo agent logs (/home/bamboo/bamboo-elastic-agent.out), I couldn't find any reason why the agent went offline!!!
Do you know where I can see the real reason for this disconnection?
On the other hand, I can see some error information in the instances list (/admin/elastic/manageElasticInstances.action).
(See screenshot || name: Bamboo_agents_information.png)
As you can see, the information is duplicated but it's wrong. It shows that 5 builds (they have the same ProjectKey-planKey, so I understand they are the same build).
They are running in the same bamboo agentID but in different EC2 instancesID.
Having a look at the bamboo server logs, I can see always these kinds of errors:
2023-01-27 02:58:04,162 INFO [ActiveMQ Session Task-73] [RemoteAgentManagerImpl] Registration attempt rejected with a message: Elastic agent with id 317358135 doesn't exist
2023-01-27 02:58:04,163 INFO [ActiveMQ Session Task-73] [ElasticInstanceManagerImpl] Registration of elastic agent at instance i-0a86f1bc7ef5c7ac1 failed with a message: Elastic agent with id 317358135 doesn't exist
2023-01-27 02:58:04,163 WARN [ActiveMQ Session Task-73] [RemoteInvocationTraceInterceptor] Processing of BambooJmsInvokerServiceExporter remote call resulted in fatal exception: com.atlassian.bamboo.buildqueue.manager.RemotedRemoteAgentManager.registerAgent
java.lang.IllegalArgumentException: Elastic agent with id 317358135 doesn't exist
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl$2.visitElastic(RemoteAgentManagerImpl.java:189) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at com.atlassian.bamboo.v2.build.agent.ElasticAgentDefinitionImpl.accept(ElasticAgentDefinitionImpl.java:99) ~[atlassian-bamboo-core-9.1.0.jar:?]
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.registerReturningAgent(RemoteAgentManagerImpl.java:182) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.registerAgent(RemoteAgentManagerImpl.java:133) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.registerAgent(RemoteAgentManagerImpl.java:119) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at sun.reflect.GeneratedMethodAccessor2401.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.24.jar:5.3.24]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.24.jar:5.3.24]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.24.jar:5.3.24]
at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:80) ~[spring-context-5.3.24.jar:5.3.24]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.24.jar:5.3.24]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.24.jar:5.3.24]
at com.sun.proxy.$Proxy321.registerAgent(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor2400.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:215) ~[spring-context-5.3.24.jar:5.3.24]
at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:39) ~[spring-context-5.3.24.jar:5.3.24]
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78) ~[spring-context-5.3.24.jar:5.3.24]
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114) ~[spring-context-5.3.24.jar:5.3.24]
at org.springframework.jms.remoting.JmsInvokerServiceExporter.onMessage(JmsInvokerServiceExporter.java:106) ~[spring-jms-5.3.24.jar:5.3.24]
at com.atlassian.bamboo.remoting.BambooJmsInvokerServiceExporter.onMessage(BambooJmsInvokerServiceExporter.java:17) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at com.atlassian.bamboo.spring.MessageListenerUtils$1.lambda$onMessage$0(MessageListenerUtils.java:25) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at com.atlassian.bamboo.spring.MessageListenerUtils.wrapWithServerFingerprintCheck(MessageListenerUtils.java:64) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at com.atlassian.bamboo.spring.MessageListenerUtils.access$000(MessageListenerUtils.java:17) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at com.atlassian.bamboo.spring.MessageListenerUtils$1.onMessage(MessageListenerUtils.java:26) ~[atlassian-bamboo-agent-core-9.1.0.jar:?]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736) ~[spring-jms-5.3.24.jar:5.3.24]
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696) ~[spring-jms-5.3.24.jar:5.3.24]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674) ~[spring-jms-5.3.24.jar:5.3.24]
at org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:645) ~[spring-jms-5.3.24.jar:5.3.24]
at org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:345) ~[spring-jms-5.3.24.jar:5.3.24]
at org.springframework.jms.listener.SimpleMessageListenerContainer.lambda$createListenerConsumer$2(SimpleMessageListenerContainer.java:322) ~[spring-jms-5.3.24.jar:5.3.24]
at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1435) ~[activemq-client-5.16.5.jar:5.16.5]
at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131) ~[activemq-client-5.16.5.jar:5.16.5]
at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202) ~[activemq-client-5.16.5.jar:5.16.5]
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133) ~[activemq-client-5.16.5.jar:5.16.5]
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) ~[activemq-client-5.16.5.jar:5.16.5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
Could you help me, please?
Thanks in advance,
Marcos Gabilondo.