Bamboo Elastic Agents Stuck in Pending

Hello,

My Bamboo setup has been broken ever since yesterday (25th Oct 2016) during the scheduled maintenance for Jira.

My Elastic Bamboo agents are forever stuck in Pending status and none of my builds can progress.

 

The elastic agents seem to come up with the following errors where user-data is not reachable or aws authentication fails when I attempt to start the agent:

```

[ec2-user@bamboo-feb-2015 ~]$ su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo

Starting Elastic Bamboo Agent...

Exception in thread "main" java.io.FileNotFoundException: This program must be run within an EC2 instance started from Bamboo: java.io.FileNotFoundException: http://169.254.169.254/2008-02-01/user-data

 at com.atlassian.aws.utils.URLFetcherUtils.fetchData(URLFetcherUtils.java:52)

 at com.atlassian.aws.utils.URLFetcherUtils.fetchString(URLFetcherUtils.java:40)

 at com.atlassian.aws.ec2.Ec2UtilsThin.getUserData(Ec2UtilsThin.java:47)

 at com.atlassian.aws.ec2.EC2Utils.getUserData(EC2Utils.java:89)

 at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.<init>(ElasticAgentInstaller.java:45)

 at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:65)

```


or 

 

```

[root@bamboo-feb-2015 ]# su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo

Starting Elastic Bamboo Agent...

2016-10-26 14:47:45,919 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-ap-se2/5.12.4-D20160704T134120/aa02e30788f7683b8f5431d112a3a6dec65b44065b68d45b46c102a9452b0435/ to /opt/bamboo-elastic-agent

2016-10-26 14:47:46,431 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-ap-se2/5.12.4-D20160704T134120/aa02e30788f7683b8f5431d112a3a6dec65b44065b68d45b46c102a9452b0435/ to /opt/bamboo-elastic-agent

2016-10-26 14:47:46,432 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-ap-se2]

2016-10-26 14:47:46,757 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-ap-se2], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 99AC87A327C26D7E), S3 Extended Request ID: Q/u6rusXcd2PbwJh9rjcW77f3iVj5I+XNx+R+AEz2vuZZ92vZHWCfCTNIHz9f/c0a1/3Hx8tFmc=

2016-10-26 14:47:46,757 INFO [main] [S3Utils] Fetching the list of remote objects...

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 981D011696993749), S3 Extended Request ID: gTbXq8/RbI5PFy8oADxXEHoJsAVEqSFM2yg6JttmGufgv102cgsvrYlztVnL0G4n2z+Q3fGl8C8=

 at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)

 at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)

 at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)

 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3714)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3664)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:637)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)

 at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:354)

 at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:162)

 at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)

  at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

```

Ive tried running latest updates. no luck. searching for the aws authentication errors due to requiring a valid date or x-amz-date header, showed a lot of issues >1 year ago with joda-time version and aws sdk.. not sure of the relation, but seems to be the same error. i have attempted to use latest joda-time but no luck.


Additionally, I can connect to the tunnel port 26224 fine.


I am really at a loss with this one.....

2 answers

This widget could not be displayed.

Is this a custom image? Which version of Bamboo is it?

This widget could not be displayed.

I had this issue when the agent was starting too early. Using systemd on CentOS 7, I resolved it by adding the following to the service unit:

[Unit]
After=network.target cloud-config.target

 This makes sure that cloud-init and the network is ready before the bamboo agent starts.

Suggest an answer

Log in or Sign up to answer
Atlassian Summit 2018

Meet the community IRL

Atlassian Summit is an excellent opportunity for in-person support, training, and networking.

Learn more
Community showcase
Posted yesterday in New to Jira

Are you planning to trial, or are currently trialling Jira Software? - We want to talk to you!

Hello! I'm Rayen, a product manager at Atlassian. My team and I are working hard to improve the trial experience for Jira Software Cloud. We are interested in   talking to 20 people planning t...

63 views 1 0
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you