New Elastic Bamboo agent stuck pending - AmazonS3Exception in log

[Dentally Portal]

Hi Team,

I've just setup a new custom elastic windows image on amazon EC2.

The connections are working ok and the image has been loaded onto a server, but when I start the agent from within bamboo, the elastic agent gets stuck on "pending".

I've logged onto the instance to take a look at the bamboo-elastic-agent.out log file, and can see the following error. Do I need to open an additional port?

JAVA_HOME: C:\Program Files\Java\jdk1.7.0_40 
PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Git\cmd 
HOME: \Users\Bamboo 
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
2013-09-19 17:13:56,773 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w2/5.0.2-OD-2/a564255d62b4cb45e6b8e2acebe7358660f32b30/ to /opt/bamboo-elastic-agent
2013-09-19 17:14:02,217 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w2/5.0.2-OD-2/a564255d62b4cb45e6b8e2acebe7358660f32b30/ to \opt\bamboo-elastic-agent
2013-09-19 17:14:02,217 INFO [main] [S3Utils] Fetching the list of remote objects...
Exception in thread "main" AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 0CB2B4F164F533B0, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: wKkYmQcytOOmP2p5hQC7uiXeNBDZsUPdbFa2VAH3GXUg/+AOxPkpIppcQxSgNukX
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2974)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2945)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)
	at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:331)
	at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:159)
	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

[Richard Barnett]

This has just started to happen for us.

Do you need to do another refresh?

[Wayne Allan]

I'm getting the same error message.

bamboo-elastic-agent.log

JAVA_HOME: C:\Program Files\Java\jdk1.7.0_40 
PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\Program Files (x86)\Git\cmd;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\NuGet;%JAVA_HOME\bin% 
HOME: \Users\Bamboo 
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
2013-09-30 00:44:47,767 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-ap-se2/5.2-OD-2/fa5fd0da044fad69f7648355dce5c5dc8f613e65/ to /opt/bamboo-elastic-agent
2013-09-30 00:44:50,903 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-ap-se2/5.2-OD-2/fa5fd0da044fad69f7648355dce5c5dc8f613e65/ to \opt\bamboo-elastic-agent
2013-09-30 00:44:50,903 INFO [main] [S3Utils] Fetching the list of remote objects...
Exception in thread "main" AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: F61A967F320E768F, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: u2Tq0QiuNyYayZKX5ncnnNLMj7xU4j3XnlVN9kNVNu3lpk/pjtbn+u1ptVvqXy51
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2974)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2945)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)
	at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:331)
	at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:159)
	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

I raised a bug for it https://jira.atlassian.com/browse/BAM-13826

I'm only guessing but is it that the automated deployments of the new agents aren't setting the right permissions on the s3 buckets?

[Dentally Portal]

Thanks - I'm now getting an error from within Bamboo on Demand itself when starting the instance - so I can't yet determine if your change has fixed my issue.

I'm getting "Failed to start elastic instances: Error when starting a new instance".

The System errors page in Bamboo contains no errors. Would this be in the log? How do I get at the Bamboo log using On Demand?

[Przemek Bruski]

Please contact support, they'll get the log and probably get back to devs to improve that error message.

[Thivagar N]

"failed to start elastic instances error when starting a new instance"

Am getting this error.. please help me

[Przemek Bruski]

I've refreshed the bucket policies, can you try again?