bamboo elastic agent permission denied on S3 sync

Jeff Behl December 12, 2015

Seen a number of posts about this sort of issue...don't get why it seems to happen regularly.  Atlassian - looks like something you need to fix?  from us-west-1

 

[root@ip-10-35-12-70 bamboo]# /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent
Syncing Elastic Bamboo Agent files...
2015-12-13 04:30:22,812 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 04:30:24,521 INFO [main] [S3Synchroniser] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 04:30:24,536 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]
2015-12-13 04:30:25,812 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 0551DDC8AC66E08A), S3 Extended Request ID: vufOZs1Jw7DKEKq2bwAT6bnE/ZUPQFdI+7jtK7ITL9jLTCuV9GhBZwUXvPa6Q8TCLrSUvhc+7lc=
2015-12-13 04:30:25,812 INFO [main] [S3Synchroniser] Fetching the list of remote objects...
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: B8E6EC6AF92964D7), S3 Extended Request ID: sic01W4ilb9JpvxXGb/hpsM6oBtqNjc7fmRwCildB9PLnKgMYHJvOvwviiuyySv0Pxo0E8+K0fA=
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3604)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3557)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:632)
	at com.atlassian.aws.s3.S3Synchroniser.getObjectNamesAndHashes(S3Synchroniser.java:341)
	at com.atlassian.aws.s3.S3Synchroniser.sync(S3Synchroniser.java:163)
	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.install(ElasticAgentInstaller.java:76)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:173)

4 answers

1 accepted

1 vote
Answer accepted
Jeff Behl December 21, 2015

So here's the answer:  Atlassian's S3 bucket that the elastic agent uses to sync some jars and whatnot is restricted to EC2 IP addresses.  Our VPC traffic runs through a VPN link and exits through our data center, meaning the source appeared to be non-EC2.  The workaround was to enable EC2 VPN endpoints which allows for internal VPC hosts to access S3 directly:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html

0 votes
Jeff Behl December 13, 2015

Apologies with the conversation with myself, but a few more data points...

The problem seems to reside round the detection of the region, perhaps due to the custom image running from inside of a VPC?  Running a stock Ubuntu image loads fine:

2015-12-13 20:44:47,248 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 20:44:51,092 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 20:44:51,092 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]
2015-12-13 20:44:53,079 INFO [main] [AmazonClients] Set S3 endpoint to: s3-us-west-1.amazonaws.com
2015-12-13 20:44:53,079 INFO [main] [S3Utils] Fetching the list of remote objects...

It should be noted it is running outside of a VPC and has a public address.

The custom image I've created and started via bamboo resides inside of a VPC and does not have a public IP address.  It is able to talk to the internet, though through a VPN connection.  It difference seems to be:

2015-12-13 04:30:25,812 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default.

Any way I can override the S3 endpoint?  Or influence the "detecting bucket location" ?

0 votes
Jeff Behl December 13, 2015

@Przemyslaw Bruski - here's the full (just tried).  And I should have mentioned this earlier:  this is a custom image that I've created to do this.  The instance was started by bamboo, however, but sits there at the "Pending" state from the Bamboo UI.  The below is after I've logged into this instance to see what could have gone wrong and tried running the bamboo-elastic-agent manually.  It's also in the "/home/bamboo/bamboo-elastic-agent.out" file, so I'm assuming it's running correctly, just getting that error and failing..

 

[root@ip-10-35-12-72 bin]# ./bamboo-elastic-agent
Syncing Elastic Bamboo Agent files...
2015-12-13 19:23:04,364 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 19:23:05,885 INFO [main] [S3Synchroniser] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent
2015-12-13 19:23:05,886 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]
2015-12-13 19:23:07,125 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1061719D3D6F6E92), S3 Extended Request ID: 1V4qvBwT0neOf+Xtd7avXigpx5XkvO4et33OVQtCEzLpcJFjOQB8+YfNXoLjM1uAtEl5in8yHII=
2015-12-13 19:23:07,125 INFO [main] [S3Synchroniser] Fetching the list of remote objects...
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5CA1ECA6CA63A5FF), S3 Extended Request ID: 00pQkCl5ITT5AGCqrQ1krQInpbXh4ofnc7R5XWxoWk6bR3MXAwoYK4cRhdnW/rZ25+GiatrvcO0=
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
..
..
0 votes
Przemek Bruski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
December 13, 2015

Is it still happening?

Jeff Behl December 13, 2015

yep...just tried: 2015-12-13 19:23:07,125 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1061719D3D6F6E92), S3 Extended Request ID: 1V4qvBwT0neOf+Xtd7avXigpx5XkvO4et33OVQtCEzLpcJFjOQB8+YfNXoLjM1uAtEl5in8yHII= 2015-12-13 19:23:07,125 INFO [main] [S3Synchroniser] Fetching the list of remote objects... Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5CA1ECA6CA63A5FF), S3 Extended Request ID: 00pQkCl5ITT5AGCqrQ1krQInpbXh4ofnc7R5XWxoWk6bR3MXAwoYK4cRhdnW/rZ25+GiatrvcO0=

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events