nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

URGENT - Elastic Bamboo Instances always stuck on 'Pending'

Rohan Eldridge

My Elastic Bamboo instances never transition from 'Pending'. What then happens is that the auto balancer kills the instance and restarts it every ~10 minutes or so! Where can I find the logs to diagnose this problem? I have pasted some of my logs at the bottom of this post but they provide zero information to help with solving this problem. Where can I find more useful logs or how can I work out what the problem is?

I have the following simple instance startup script in my Elastic Image configuration:

sudo yum install -y php-mysql

sudo yum install -y php-mysqli

sudo yum install -y php-pdo

sudo yum install -y php-pecl-xdebug

I originally had more commands in the instance startup script but removed them to try and resolve this problem.

The AMI ID of my Elastic Image is ami-430a452a. The name of the Elastic Image is "Amazon Linux: Default Image S3 x86_64 (linux) (stock image)".

Here is all that I get in the logs:

2013-11-10 02:19:19,855 INFO [main] [AgentContext] Bootstrap Type: 'ELASTIC'

2013-11-10 02:19:19,963 INFO [Thread-2] [RemoteAgent] com.atlassian.bamboo.agent.elastic.client.ElasticAgent starting.

2013-11-10 02:19:20,231 INFO [Thread-2] [RemoteAgentHomeLocatorForBootstrap] Agent home located at '/home/bamboo/bamboo-agent-home'

2013-11-10 02:19:20,239 INFO [Thread-2] [RemoteAgent] Starting Bamboo Remote Agent...

2013-11-10 02:19:20,240 INFO [Thread-2] [RemoteAgent] Starting application context...

2013-11-10 02:19:22,326 WARN [Thread-2] [PropertyPlaceholderConfigurer] Could not load properties from class path resource [bamboo-init.properties]: class path resource [bamboo-init.properties] cannot be opened because it does not exist

2013-11-10 02:19:22,367 INFO [Thread-2] [RemoteAgentHomeLocatorForBootstrap] Agent home located at '/home/bamboo/bamboo-agent-home'

2013-11-10 02:19:22,368 INFO [Thread-2] [RemoteAgentHomeDirectoryLayout] Plugin directory: /home/bamboo/bamboo-agent-home/plugins

2013-11-10 02:19:22,368 INFO [Thread-2] [RemoteAgentHomeDirectoryLayout] Plugin cache directory: /home/bamboo/bamboo-agent-home/caches/plugins

2013-11-10 02:19:22,410 WARN [Thread-2] [AgentConfiguration] Configuration file does not exist. Creating File: bamboo-agent.cfg.xml

2013-11-10 02:19:22,524 INFO [Thread-2] [AgentUuidInitializer] Found agent UUID '3d1fc27e-c6c9-4805-a4b9-c295097134cd' in temporary UUID file '/home/bamboo/bamboo-agent-home/uuid-temp.properties'

2013-11-10 02:19:22,524 INFO [Thread-2] [AgentConfiguration] Setting UUID in main config file to '3d1fc27e-c6c9-4805-a4b9-c295097134cd'

2013-11-10 02:19:22,580 INFO [Thread-2] [BambooActiveMQConnectionFactory] Setting broker URL to 'failover:(tcp://localhost:4527?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&initialReconnectDelay=15000'

2013-11-10 02:19:22,728 INFO [Thread-2] [BambooActiveMQConnectionFactory] Setting broker URL to 'failover:(tcp://localhost:4527?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&initialReconnectDelay=15000'

2013-11-10 02:19:23,121 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-3] Accepted a local connection on port 4527, tunnel age: 7s

2013-11-10 02:19:33,141 WARN [Thread-2] [FailoverTransport] Transport (tcp://127.0.0.1:4527) failed, reason: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format., attempting to automatically reconnect

2013-11-10 02:19:48,142 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-4] Accepted a local connection on port 4527, tunnel age: 24s

2013-11-10 02:20:43,093 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-3] Accepted a local connection on port 4527, tunnel age: 54s

Accepted answers

All comments

Przemek Bruski

It basically means that the agent was unable to contact the server. Does it work if you don't use an instance setup script?

Rohan Eldridge

No, it doesn't work even if I remove the instance setup script. It still stays stuck on 'Pending'.

Am I using an old/wrong Bamboo EC2 agent image?

Przemek Bruski

No, it does not look like a wrong image. Do you use a VPC?

Rohan Eldridge

According to the EC2 console, no Elastic IPs are currently in use in our Bamboo Elastic Agent region (ie. US East). There is also no VPC set up in that region. Any idea as to why the Agents aren't being given Elastic IPs then as I assumed that was a automatic process?

I shall look into the Elastic IP management as well.

Steffen Opel

@Rohan - you can check whether you image configuration is in a VPC by going to 'Bamboo admin -> Elastic Bamboo -> Image Configurations' and clicking 'Edit' on the image(s) currently in use. The combo box '<label id="fieldLabelArea_saveElasticImageConfiguration_vpc" for="saveElasticImageConfiguration_vpc">Virtual private cloud subnet</label>' needs to be set to 'None' rather than a VPC subnet, if any (see Managing your elastic image configurations for details).

That being said, I doubt it will be the problem, so a few more questions in case:

Have your agents always been running in that particular AWSaccount in us-east-1, i.e. they have neither been migrated to a different region or AWS account?
- Regardless, is this a comparatively new AWS account? In case, could you confirm via the AWS Management Console, whether you are running on the new 'EC2-VPC' platform or can use both 'EC2' and 'VPC' separately (section 'Supported Platforms' in the top right of the EC2 dashboard)?
- In case you are running on the new 'EC2-VPC' platform, is this AWS account used for other activities too, i.e. is it possible that somebody changed things in that account?

razbakov

I have the same issue.

My Elastic agent is Pending

Krystian Brazulewicz

I was able to replicate this problem. In order to fix it I needed to add the following line to /etc/hosts on the Bamboo server instance:

10.0.1.113  ip-10-0-1-113

please adjust to your hostname/IP address. This change is related to the fact that bare EC2 instance is unable to resolve its own name:

ec2-user@ip-10-0-1-113 bamboo-home]$ hostname
ip-10-0-1-113
[ec2-user@ip-10-0-1-113 bamboo-home]$ host ip-10-0-1-113
Host ip-10-0-1-113 not found: 3(NXDOMAIN)
[ec2-user@ip-10-0-1-113 bamboo-home]$ host ip-10-0-1-113.ec2.internal
ip-10-0-1-113.ec2.internal has address 10.0.1.113

mfobrien NA

I am having the same issue launching a bamboo EC2 instance inside a private subnet on my VPC under 172.21.1.0. I have a 2 zone VPC with NATs running on the public 0 subnet.

I am able to connect to the instance only via a bastion running on the public subnet by tunnelling to the private bamboo instance.

I checked that the above hostname issue is fixed as of Jan 2015

ec2-user@ip-172-21-1-46 ~$ hostname
ip-172-21-1-46
ec2-user@ip-172-21-1-46 ~$ host ip-172-21-1-46
ip-172-21-1-46.ec2.internal has address 172.21.1.46

I manually associated an EIP (from the VPC console) and this did not help - so the issue is not EIP - because a non-VPC 172.31 (amazon default) public EC2 instance works fine with just a public IP and no EIP.

ports are fine 22 and

tcp 0 0 :::26224 :::* LISTEN -

Running on the public 172.21.0.0/26 subnet works via the public IP - but this is not a secure workaround

see related

Bamboo can't start Amazon Elastic Instance agent after being added to VPC

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

cloud

bamboo