bamboo elastic agent permission denied on S3 sync

Seen a number of posts about this sort of issue...don't get why it seems to happen regularly.  Atlassian - looks like something you need to fix?  from us-west-1

 

[root@ip-10-35-12-70 bamboo]# /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent
Syncing Elastic Bamboo Agent files...
2015-12-13 04:30:22,812 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 04:30:24,521 INFO [main] [S3Synchroniser] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 04:30:24,536 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]
2015-12-13 04:30:25,812 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 0551DDC8AC66E08A), S3 Extended Request ID: vufOZs1Jw7DKEKq2bwAT6bnE/ZUPQFdI+7jtK7ITL9jLTCuV9GhBZwUXvPa6Q8TCLrSUvhc+7lc=
2015-12-13 04:30:25,812 INFO [main] [S3Synchroniser] Fetching the list of remote objects...
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: B8E6EC6AF92964D7), S3 Extended Request ID: sic01W4ilb9JpvxXGb/hpsM6oBtqNjc7fmRwCildB9PLnKgMYHJvOvwviiuyySv0Pxo0E8+K0fA=
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3604)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3557)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:632)
	at com.atlassian.aws.s3.S3Synchroniser.getObjectNamesAndHashes(S3Synchroniser.java:341)
	at com.atlassian.aws.s3.S3Synchroniser.sync(S3Synchroniser.java:163)
	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.install(ElasticAgentInstaller.java:76)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:173)

4 answers

1 accepted

So here's the answer:  Atlassian's S3 bucket that the elastic agent uses to sync some jars and whatnot is restricted to EC2 IP addresses.  Our VPC traffic runs through a VPN link and exits through our data center, meaning the source appeared to be non-EC2.  The workaround was to enable EC2 VPN endpoints which allows for internal VPC hosts to access S3 directly:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html

0 vote

Is it still happening?

yep...just tried: 2015-12-13 19:23:07,125 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1061719D3D6F6E92), S3 Extended Request ID: 1V4qvBwT0neOf+Xtd7avXigpx5XkvO4et33OVQtCEzLpcJFjOQB8+YfNXoLjM1uAtEl5in8yHII= 2015-12-13 19:23:07,125 INFO [main] [S3Synchroniser] Fetching the list of remote objects... Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5CA1ECA6CA63A5FF), S3 Extended Request ID: 00pQkCl5ITT5AGCqrQ1krQInpbXh4ofnc7R5XWxoWk6bR3MXAwoYK4cRhdnW/rZ25+GiatrvcO0=

@Przemyslaw Bruski - here's the full (just tried).  And I should have mentioned this earlier:  this is a custom image that I've created to do this.  The instance was started by bamboo, however, but sits there at the "Pending" state from the Bamboo UI.  The below is after I've logged into this instance to see what could have gone wrong and tried running the bamboo-elastic-agent manually.  It's also in the "/home/bamboo/bamboo-elastic-agent.out" file, so I'm assuming it's running correctly, just getting that error and failing..

 

[root@ip-10-35-12-72 bin]# ./bamboo-elastic-agent
Syncing Elastic Bamboo Agent files...
2015-12-13 19:23:04,364 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 19:23:05,885 INFO [main] [S3Synchroniser] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 19:23:05,886 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]
2015-12-13 19:23:07,125 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1061719D3D6F6E92), S3 Extended Request ID: 1V4qvBwT0neOf+Xtd7avXigpx5XkvO4et33OVQtCEzLpcJFjOQB8+YfNXoLjM1uAtEl5in8yHII=
2015-12-13 19:23:07,125 INFO [main] [S3Synchroniser] Fetching the list of remote objects...
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5CA1ECA6CA63A5FF), S3 Extended Request ID: 00pQkCl5ITT5AGCqrQ1krQInpbXh4ofnc7R5XWxoWk6bR3MXAwoYK4cRhdnW/rZ25+GiatrvcO0=
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
..
..

Apologies with the conversation with myself, but a few more data points...

The problem seems to reside round the detection of the region, perhaps due to the custom image running from inside of a VPC?  Running a stock Ubuntu image loads fine:

2015-12-13 20:44:47,248 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 20:44:51,092 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 20:44:51,092 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]
2015-12-13 20:44:53,079 INFO [main] [AmazonClients] Set S3 endpoint to: s3-us-west-1.amazonaws.com
2015-12-13 20:44:53,079 INFO [main] [S3Utils] Fetching the list of remote objects...

It should be noted it is running outside of a VPC and has a public address.

The custom image I've created and started via bamboo resides inside of a VPC and does not have a public IP address.  It is able to talk to the internet, though through a VPN connection.  It difference seems to be:

2015-12-13 04:30:25,812 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default.

Any way I can override the S3 endpoint?  Or influence the "detecting bucket location" ?

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published May 18, 2017 in Bamboo

FAQ: How to Upgrade Bamboo Server

Bamboo 5.9 will no longer be supported after June 12, 2017. What does this mean? As part of our End of Life policy, Atlassian supports major versions for two years after the first major iteratio...

1,817 views 0 6
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you