My Elastic Bamboo instances never transition from 'Pending'. What then happens is that the auto balancer kills the instance and restarts it every ~10 minutes or so! Where can I find the logs to diagnose this problem? I have pasted some of my logs at the bottom of this post but they provide zero information to help with solving this problem. Where can I find more useful logs or how can I work out what the problem is?
I have the following simple instance startup script in my Elastic Image configuration:
sudo yum install -y php-mysql
sudo yum install -y php-mysqli
sudo yum install -y php-pdo
sudo yum install -y php-pecl-xdebug
I originally had more commands in the instance startup script but removed them to try and resolve this problem.
The AMI ID of my Elastic Image is ami-430a452a. The name of the Elastic Image is "Amazon Linux: Default Image S3 x86_64 (linux) (stock image)".
Here is all that I get in the logs:
2013-11-10 02:19:19,855 INFO [main] [AgentContext] Bootstrap Type: 'ELASTIC'
2013-11-10 02:19:19,963 INFO [Thread-2] [RemoteAgent] com.atlassian.bamboo.agent.elastic.client.ElasticAgent starting.
2013-11-10 02:19:20,231 INFO [Thread-2] [RemoteAgentHomeLocatorForBootstrap] Agent home located at '/home/bamboo/bamboo-agent-home'
2013-11-10 02:19:20,239 INFO [Thread-2] [RemoteAgent] Starting Bamboo Remote Agent...
2013-11-10 02:19:20,240 INFO [Thread-2] [RemoteAgent] Starting application context...
2013-11-10 02:19:22,326 WARN [Thread-2] [PropertyPlaceholderConfigurer] Could not load properties from class path resource [bamboo-init.properties]: class path resource [bamboo-init.properties] cannot be opened because it does not exist
2013-11-10 02:19:22,367 INFO [Thread-2] [RemoteAgentHomeLocatorForBootstrap] Agent home located at '/home/bamboo/bamboo-agent-home'
2013-11-10 02:19:22,368 INFO [Thread-2] [RemoteAgentHomeDirectoryLayout] Plugin directory: /home/bamboo/bamboo-agent-home/plugins
2013-11-10 02:19:22,368 INFO [Thread-2] [RemoteAgentHomeDirectoryLayout] Plugin cache directory: /home/bamboo/bamboo-agent-home/caches/plugins
2013-11-10 02:19:22,410 WARN [Thread-2] [AgentConfiguration] Configuration file does not exist. Creating File: bamboo-agent.cfg.xml
2013-11-10 02:19:22,524 INFO [Thread-2] [AgentUuidInitializer] Found agent UUID '3d1fc27e-c6c9-4805-a4b9-c295097134cd' in temporary UUID file '/home/bamboo/bamboo-agent-home/uuid-temp.properties'
2013-11-10 02:19:22,524 INFO [Thread-2] [AgentConfiguration] Setting UUID in main config file to '3d1fc27e-c6c9-4805-a4b9-c295097134cd'
2013-11-10 02:19:22,580 INFO [Thread-2] [BambooActiveMQConnectionFactory] Setting broker URL to 'failover:(tcp://localhost:4527?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&initialReconnectDelay=15000'
2013-11-10 02:19:22,728 INFO [Thread-2] [BambooActiveMQConnectionFactory] Setting broker URL to 'failover:(tcp://localhost:4527?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&initialReconnectDelay=15000'
2013-11-10 02:19:23,121 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-3] Accepted a local connection on port 4527, tunnel age: 7s
2013-11-10 02:19:33,141 WARN [Thread-2] [FailoverTransport] Transport (tcp://127.0.0.1:4527) failed, reason: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format., attempting to automatically reconnect
2013-11-10 02:19:48,142 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-4] Accepted a local connection on port 4527, tunnel age: 24s
2013-11-10 02:20:43,093 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-3] Accepted a local connection on port 4527, tunnel age: 54s
According to the EC2 console, no Elastic IPs are currently in use in our Bamboo Elastic Agent region (ie. US East). There is also no VPC set up in that region. Any idea as to why the Agents aren't being given Elastic IPs then as I assumed that was a automatic process?
I shall look into the Elastic IP management as well.
I believe we are using a VPC for the actual application that we have running on our AWS account - it has public and private subnets. I just checked the Bamboo Elastic Agent Instance and it doesn't have an Elastic IP, which I assume means it can't talk to the outside world correctly?
Is there a guide or documentation for checking our AWS setup to ensure it'll work with Bamboo?
I know recently we did run into a problem of running out of Elastic IPs, we did request more from AWS however and they gave them to us.
Whether you have VPCs enabled on that account doesn't really matter - what matters is whether your image configuration is configured to use VPC.
BTW. you don't have to have EIPs allocated if you enabled EIP management - Bamboo will allocate and deallocate them.
I'm guessing that "Enable elastic IP management" will fix the problem... so I guess my new question is, why do I need to do this now when before we didn't need to modify this setting?
It's not the end of the world if we do need to enable that setting, I'm just curious as we didn't need to enable that setting before and we had Elastic Agents working correctly. I'm just trying to understand why things have changed and what we could have done wrong.
OK, looking at the original image it's not customised, it seems to be based on a stock image, which is ami-a10b44c8. It does have a snapshot attached but I don't think that would affect how it interacts with the network?!?
As a result my assumption would be that the instances have always needed Elastic IPs and for some reason either a Bamboo setting was changed or the Elastic IPs were manually allocated at the time and then removed later.
I guess enabling "elastic ip management" will fix things, but our Bamboo admin doesn't want me to do that...
Anyway, thanks for your help - and please correct me if any of my conclusions are wrong.
Sorry for all the questions, I'm just confused at to why it has stopped working when it was working. As a result I'm trying to figure out what has changed and if it is a Bamboo setting OR a problem with our AWS.
So it's not just a case of how to fix it, it's also a case of figuring out why it broke :)
@Rohan - you can check whether you image configuration is in a VPC by going to 'Bamboo admin -> Elastic Bamboo -> Image Configurations' and clicking 'Edit' on the image(s) currently in use. The combo box Virtual private cloud subnet needs to be set to None rather than a VPC subnet, if any (see Managing your elastic image configurations for details).
That being said, I doubt it will be the problem, so a few more questions in case:
Yeah, I have no option in that drop down to pick a subnet.
1. We've always run them in US East as we're using all the original images. As for availability zone, that has always been left on automatic.
2. We have EC2 and VPC seperate under 'Supported Platforms'.
3. Someone may have changed things on the account that I'm unaware of, but from what I can tell there is no VPC or Elastic IPs in use currently in the region. The only thing in the region is an Elastic Beanstalk based EC2 app.
@Rohan - you can check whether you image configuration is in a VPC by going to 'Bamboo admin -> Elastic Bamboo -> Image Configurations' and clicking 'Edit' on the image(s) currently in use. The combo box '<label id="fieldLabelArea_saveElasticImageConfiguration_vpc" for="saveElasticImageConfiguration_vpc">Virtual private cloud subnet</label>' needs to be set to 'None' rather than a VPC subnet, if any (see Managing your elastic image configurations for details).
That being said, I doubt it will be the problem, so a few more questions in case:
I was able to replicate this problem. In order to fix it I needed to add the following line to /etc/hosts on the Bamboo server instance:
please adjust to your hostname/IP address. This change is related to the fact that bare EC2 instance is unable to resolve its own name:
ec2-user@ip-10-0-1-113 bamboo-home]$ hostname ip-10-0-1-113 [ec2-user@ip-10-0-1-113 bamboo-home]$ host ip-10-0-1-113 Host ip-10-0-1-113 not found: 3(NXDOMAIN) [ec2-user@ip-10-0-1-113 bamboo-home]$ host ip-10-0-1-113.ec2.internal ip-10-0-1-113.ec2.internal has address 10.0.1.113
I am having the same issue launching a bamboo EC2 instance inside a private subnet on my VPC under 172.21.1.0. I have a 2 zone VPC with NATs running on the public 0 subnet.
I am able to connect to the instance only via a bastion running on the public subnet by tunnelling to the private bamboo instance.
I checked that the above hostname issue is fixed as of Jan 2015
I manually associated an EIP (from the VPC console) and this did not help - so the issue is not EIP - because a non-VPC 172.31 (amazon default) public EC2 instance works fine with just a public IP and no EIP.
ports are fine 22 and
tcp 0 0 :::26224 :::* LISTEN -
Running on the public 172.21.0.0/26 subnet works via the public IP - but this is not a secure workaround
This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.Read more
Bamboo 5.9 will no longer be supported after June 12, 2017. What does this mean? As part of our End of Life policy, Atlassian supports major versions for two years after the first major iteratio...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs