Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Bamboo Elastic Agents Stuck in Pending

Hello,

My Bamboo setup has been broken ever since yesterday (25th Oct 2016) during the scheduled maintenance for Jira.

My Elastic Bamboo agents are forever stuck in Pending status and none of my builds can progress.

 

The elastic agents seem to come up with the following errors where user-data is not reachable or aws authentication fails when I attempt to start the agent:

```

[ec2-user@bamboo-feb-2015 ~]$ su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo

Starting Elastic Bamboo Agent...

Exception in thread "main" java.io.FileNotFoundException: This program must be run within an EC2 instance started from Bamboo: java.io.FileNotFoundException: http://169.254.169.254/2008-02-01/user-data

 at com.atlassian.aws.utils.URLFetcherUtils.fetchData(URLFetcherUtils.java:52)

 at com.atlassian.aws.utils.URLFetcherUtils.fetchString(URLFetcherUtils.java:40)

 at com.atlassian.aws.ec2.Ec2UtilsThin.getUserData(Ec2UtilsThin.java:47)

 at com.atlassian.aws.ec2.EC2Utils.getUserData(EC2Utils.java:89)

 at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.<init>(ElasticAgentInstaller.java:45)

 at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:65)

```


or 

 

```

[root@bamboo-feb-2015 ]# su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo

Starting Elastic Bamboo Agent...

2016-10-26 14:47:45,919 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-ap-se2/5.12.4-D20160704T134120/aa02e30788f7683b8f5431d112a3a6dec65b44065b68d45b46c102a9452b0435/ to /opt/bamboo-elastic-agent

2016-10-26 14:47:46,431 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-ap-se2/5.12.4-D20160704T134120/aa02e30788f7683b8f5431d112a3a6dec65b44065b68d45b46c102a9452b0435/ to /opt/bamboo-elastic-agent

2016-10-26 14:47:46,432 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-ap-se2]

2016-10-26 14:47:46,757 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-ap-se2], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 99AC87A327C26D7E), S3 Extended Request ID: Q/u6rusXcd2PbwJh9rjcW77f3iVj5I+XNx+R+AEz2vuZZ92vZHWCfCTNIHz9f/c0a1/3Hx8tFmc=

2016-10-26 14:47:46,757 INFO [main] [S3Utils] Fetching the list of remote objects...

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 981D011696993749), S3 Extended Request ID: gTbXq8/RbI5PFy8oADxXEHoJsAVEqSFM2yg6JttmGufgv102cgsvrYlztVnL0G4n2z+Q3fGl8C8=

 at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)

 at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)

 at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)

 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3714)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3664)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:637)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)

 at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:354)

 at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:162)

 at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)

  at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

```

Ive tried running latest updates. no luck. searching for the aws authentication errors due to requiring a valid date or x-amz-date header, showed a lot of issues >1 year ago with joda-time version and aws sdk.. not sure of the relation, but seems to be the same error. i have attempted to use latest joda-time but no luck.


Additionally, I can connect to the tunnel port 26224 fine.


I am really at a loss with this one.....

2 answers

I had this issue when the agent was starting too early. Using systemd on CentOS 7, I resolved it by adding the following to the service unit:

[Unit]
After=network.target cloud-config.target

 This makes sure that cloud-init and the network is ready before the bamboo agent starts.

0 votes

Is this a custom image? Which version of Bamboo is it?

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bamboo

Bamboo Data Center 8.1 is now available

G’day Bamboo customers, Bamboo DC 8.1 is now available with it the following features and programs: SAML 2.0, OpenID Connect, and Crowd SSO In order to help admins with a simplified user manage...

204 views 0 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you