FATAL ERROR: Bamboo Remote Agent is loading constently

[Davy HANTZ]

Hello,

I'm facing this error when using Bamboo Agent Base (https://bitbucket.org/atlassian/docker-bamboo-agent-base/overview) and i have no clue about it, can someone help me find out how to fix it ?

I'm running on a dockerize bamboo server 6.8.0 and another container containing the docker-bamboo-agent-base.

When i launch, everything work, i can see the remote connecting but he is stuck in loading with this error raising.

2019-03-26 14:17:38,081 FATAL [AgentRunnerThread] [AgentBootstrap] Exiting due to fatal exception.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:26)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentRegistrationBean': Invocation of init method failed; nested exception is org.springframework.remoting.RemoteAccessException: Could not access JMS invoker queue [queue://com.atlassian.bamboo.registrationQueue]; nested exception is javax.jms.IllegalStateException: javax.jms.JMSException: d80cbb20f13d
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:139)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1691)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:144)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:95)
at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent$1.<init>(RemoteAgent.java:214)
at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:214)
at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:82)
... 6 more
Caused by: org.springframework.remoting.RemoteAccessException: Could not access JMS invoker queue [queue://com.atlassian.bamboo.registrationQueue]; nested exception is javax.jms.IllegalStateException: javax.jms.JMSException: d80cbb20f13d
at org.springframework.jms.remoting.JmsInvokerClientInterceptor.convertJmsInvokerAccessException(JmsInvokerClientInterceptor.java:432)
at org.springframework.jms.remoting.JmsInvokerClientInterceptor.invoke(JmsInvokerClientInterceptor.java:207)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy21.registerAgent(Unknown Source)
at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:309)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
... 22 more
Caused by: javax.jms.IllegalStateException: javax.jms.JMSException: d80cbb20f13d
at org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:162)
at org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
at org.springframework.jms.remoting.JmsInvokerClientInterceptor.createSession(JmsInvokerClientInterceptor.java:281)
at org.springframework.jms.remoting.JmsInvokerClientInterceptor.executeRequest(JmsInvokerClientInterceptor.java:250)
at org.springframework.jms.remoting.JmsInvokerClientInterceptor.invoke(JmsInvokerClientInterceptor.java:204)
... 33 more
Caused by: javax.jms.JMSException: d80cbb20f13d
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1413)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1478)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:329)
at org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:112)
at org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:82)
at org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:78)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:357)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:279)
at org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:144)
... 37 more
Caused by: java.net.UnknownHostException: d80cbb20f13d
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:525)
at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:488)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:169)
at org.apache.activemq.transport.InactivityMonitor.start(InactivityMonitor.java:52)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more

[Daniel Santos · Accepted solution]

Hi @Davy HANTZ

 I see you are new here, welcome to our community! =]

According to this message 

Caused by: java.net.UnknownHostException: d80cbb20f13d

I assume the issue is caused by a configuration in the Bamboo broker. Your agent is not capable of resolving the host configured for the broker.

An agent has two types of communication to reach Bamboo. The first one is through HTTP(S) the second one is thought JMS. The first one seems to be working fine for you. The agent is reaching Bamboo and Bamboo is capable to see a new agent is loading. 

I believe your agent is also capable of receiving the Bamboo answer with the broker address, but at this point is not able to reach the host from that address.

Suggestion to move forward

1. Stop Bambo

2. Please doublecheck that <Bamboo_Server_Container_Home>/bamboo.cfg.xml has the following properties configured as described below:

• bamboo.jms.broker.client.uri
  

  failover:(<RESOLVABLE_URL_OR_IP>://URL:54663?wireFormat.maxInactivityDuration=300000)?initialReconnectDelay=15000&amp;maxReconnectAttempts=10

• bamboo.jms.broker.uri
  

  nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=300000

3. Start Bamboo

4. Restart the agent

Please let us know how it goes.

[Davy HANTZ]

Ok, your answer make me realize that i didn't set correctly my docker network, i change it so now it's working fine but your solution would also work !

Thanks a lot !

[Daniel Santos]

I'm glad to help in any way!
Have a good one!

[Kevin De Wolf]

Hello @Davy HANTZ 

Could you let me know how you configured your docker network correctly? I'm running into the same problem.

Thanks!

[Davy HANTZ]

@Kevin De Wolf 

Just make sure that your remote agent and bamboo are in the same docker network. In my case i have create an "atlassian" docker network and run my Agent Remote and Bamboo in this network

[Kevin De Wolf]

@Davy HANTZ 

Thanks for the feedback!

My intention is to have Bamboo server outside of the docker network, so unfortunately this will not help for me. I did however continued my search, and found that it was caused by the broker url not being able to be resolved on the agents. So for any others having similar problems: make sure the port 54663 is opened, and the hostname in the broker url is resolvable from the agents.

[Daniel Santos]

Yes, @Kevin De Wolf that port needs to be open. You can also use a different port by changing the value in <Bamboo_Home>/bamboo.cfg.xml and restarting the Bamboo server.

The agent is the one who seeks for the Bamboo server and it will first do an HTTP request to Bamboo asking for the broker configuration and then use whatever configuration was shared by the Bamboo server.

When you change the port there, Bamboo will load on a different port and will inform the new port to any agent that is starting up. It means that when this change is performed you need also to restart the agents when the Bamboo server is up after the restart.