Bamboo Elastic Agent stuck in Pending - ActiveMQ Transport Connection Failed

[drewk]

I'm trying to connect elastic agents to my Bamboo Server for the first time and the agent is stuck in a pending state. My question is, how can I troubleshoot exactly why the agent is stuck in the pending state.

While it appears there is some communication between server and agent, the standout issue in my opinion is:

2015-11-03 10:13:19,734 WARN [ActiveMQ Transport: tcp:///x.x.x.x:47678@54663] [Transport] Transport Connection to: tcp://x.x.x.x:47678 failed: java.io.EOFException

How I've created the agent

I've taken the latest AmazonLinux image and done the following:
- updated the timezone to ensure the time matches the server
- using wget to grab the .jar directly from the server
- running java to spin up the jar/agent (ignoring the certs names just incase)

What I've checked

I've checked firewall and networking between server and agent and can't find any evidence that we're specifically blocking traffic, in fact the logs below show at least SOME/MOST connectivity to be okay.

Bamboo Log Snippet

2015-11-03 10:00:39,105 INFO [ActiveMQ Session Task-2] [RemoteAgentManagerImpl] Remote agent created with id xxxxx
2015-11-03 10:00:39,105 INFO [ActiveMQ Session Task-2] [RemoteAgentManagerImpl] Remote agent "ip-x-x-x-x.ap-southeast-2.compute.internal" has registered.
2015-11-03 10:13:19,734 WARN [ActiveMQ Transport: tcp:///x.x.x.x:47678@54663] [Transport] Transport Connection to: tcp://x.x.x.x:47678 failed: java.io.EOFException

Agent Log Snippet

INFO | jvm 1 | 2015/11/02 23:18:55 | 2015-11-02 23:18:55,108 WARN [Thread-0] [ResourceResolver] Unable to find resource bamboo-capabilities.properties
INFO | jvm 1 | 2015/11/02 23:18:55 | 2015-11-02 23:18:55,109 INFO [Thread-0] [DefaultAgentCapabilityManagerImpl] Failed to load bamboo-capabilities.properties
INFO | jvm 1 | 2015/11/02 23:18:58 | 2015-11-02 23:18:58,671 WARN [Felix:Startup] [DefaultComponentRegistrar] Host component buildDefinitionConverter of instance com.atlassian.bamboo.fieldvalue.BuildDefinitionConverter@26313223 has no interfaces
INFO | jvm 1 | 2015/11/02 23:19:09 | 2015-11-02 23:19:09,569 WARN [Thread-0] [PropertyPlaceholderConfigurer] Could not load properties from class path resource [bamboo-init.properties]: class path resource [bamboo-init.properties] cannot be opened because it does not exist
INFO | jvm 1 | 2015/11/02 23:19:09 | 2015-11-02 23:19:09,602 WARN [Thread-0] [Jdk] Environment variable JAVA_HOME does not point to valid JDK
INFO | jvm 1 | 2015/11/02 23:19:11 | 2015-11-02 23:19:11,867 INFO [Thread-0] [DefaultBuildAgent] Build agent 'ip-x-x-x-x.ap-southeast-2.compute.internal' started. Waiting for builds...
INFO | jvm 1 | 2015/11/02 23:19:11 | 2015-11-02 23:19:11,892 INFO [Thread-0] [AgentHeartBeatJobScheduler] Scheduled AgentHeartBeatJobScheduler to run every 60s. Next run at Mon Nov 02 23:19:11 UTC 2015
INFO | jvm 1 | 2015/11/02 23:19:11 | 2015-11-02 23:19:11,912 INFO [0-BAM::ip-x-x-x-x.ap-southeast-2.compute.internal::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 'ip-x-x-x-x.ap-southeast-2.compute.internal' ready to take build from queue...
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,943 INFO [0-BAM::ip-x-x-x-x.ap-southeast-2.compute.internal::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent xxxxx checking build queue for executables...
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,960 INFO [scheduler_Worker-1] [AgentHeartBeatJob] executableBuildAgent still unavailable. Heartbeat skipped.
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,972 INFO [Thread-0] [RemoteAgent] **************************************************************************************************************************************************
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,972 INFO [Thread-0] [RemoteAgent] * *
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,972 INFO [Thread-0] [RemoteAgent] * Bamboo agent 'ip-x-x-x-x.ap-southeast-2.compute.internal' ready to receive builds.
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,972 INFO [Thread-0] [RemoteAgent] * Remote Agent Home: /root/bamboo-agent-home
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,972 INFO [Thread-0] [RemoteAgent] * Broker URL: failover:(tcp://bamboo:54663?wireFormat.maxInactivityDuration=300000)?initialReconnectDelay=15000&maxReconnectAttempts=10
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,972 INFO [Thread-0] [RemoteAgent] * *
INFO | jvm 1 | 2015/11/02 23:19:12 | 2015-11-02 23:19:11,973 INFO [Thread-0] [RemoteAgent] **************************************************************************************************************************************************
INFO | jvm 1 | 2015/11/02 23:24:12 | 2015-11-02 23:24:12,237 INFO [0-BAM::ip-x-x-x-x.ap-southeast-2.compute.internal::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 2785281 checking build queue for executables...

[Przemek Bruski · Accepted solution]

You're mixing up two types of agents.

You're using a plain remote agent on an elastic image. This agent connects and works fine.

However, you've also defined that image as a host for an elastic agent, which you haven't installed on that image.

You need to follow https://confluence.atlassian.com/bamboo/creating-a-custom-elastic-image-289277146.html .

Or, use the stock images.