nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Agent fails to start due to failure to sync agent from s3

Lyndon Howie

Everything worked fine yesterday, but from today our elastic agents aren't starting up. The output in the bamboo agent log file is:

2021-11-30 00:50:34,454 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-ap-se2/5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/ to /opt/bamboo-elastic-agent
2021-11-30 00:50:36,861 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-ap-se2/5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/ to \opt\bamboo-elastic-agent
2021-11-30 00:50:36,861 INFO [main] [S3Utils] Fetching the list of remote objects...
Exception in thread "main" AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: P5G3BVK0TSS4BFSM, AWS Error Code: InvalidAccessKeyId, AWS Error Message: The AWS Access Key Id you provided does not exist in our records., S3 Extended Request ID: XYkVwZyx8rQ9npwjCI9JzbPptkIpfFEOvL+jEngN8D4GdWe8/PqUhVmmODK/c+frRyzBvi4+lpI=
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2974)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2945)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)
	at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:331)
	at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:159)
	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

I don't think there is an issue with our AWS credentials (they haven't changed since yesterday when everything was working), and as far as i'm aware they are irrelevant for this operation anyway.

We also can't access this url either from aws command line (with other credentials) or via the browser.

Have these files been removed from S3? If so can they be reinstated, or if not is there a workaround to tell bamboo to skip this sync and just use the files that are already on the instance? (or point it at our own s3 bucket or something?) not sure exactly what it syncs from there...

Help greatly appreciated!

Accepted answers

Eduardo Alvarenga

Hello @Lyndon Howie

Thank you for reaching out to Atlassian Community!

Bamboo S3 buckets have an access list that would only accept Amazon's CIDRs, meaning that any non-AWS IP addresses will not be able to retrieve any data from the buckets.

I've tested it from an ordinary Elastic Agent and got the following:

root@ip-10-116-101-158:~# aws s3 ls --recursive s3://bamboo-agent-release-ap-se2/5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef
2016-10-31 09:05:23 17536 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/annotations-13.0.jar
2016-10-31 09:05:18 10399 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-annotations-1.1.0.jar
2016-10-31 09:05:21 37954 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-aws-bootstrap-1.0.132.jar
2016-10-31 09:05:32 69118 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-agent-bootstrap-5.14.0.2.jar
2016-10-31 09:05:17 30378 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-agent-elastic-5.14.0.2.jar
2016-10-31 09:05:34 7693 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-agent-elastic-shared-5.14.0.2.jar
2016-10-31 09:05:13 3754 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-api-agent-bootstrap-5.14.0.2.jar
2016-10-31 09:05:30 5077 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-core-agent-bootstrap-5.14.0.2.jar
2016-10-31 09:05:10 49998 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-tunnel-1.8.4.jar
2016-10-31 09:05:17 729956 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-core-1.11.40.jar
2016-10-31 09:05:09 3694062 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-ec2-1.11.40.jar
2016-10-31 09:05:15 427820 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-kms-1.11.40.jar
2016-10-31 09:05:10 655656 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-s3-1.11.40-atlassian-1.jar
2016-10-31 09:05:14 3277268 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/bcprov-jdk15on-1.54.jar
2016-10-31 09:05:23 263865 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/commons-codec-1.8.jar
2016-10-31 09:05:16 60686 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/commons-logging-1.1.1.jar
2016-10-31 09:05:27 197860 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/gson-2.2.2-atlassian-1.jar
2016-10-31 09:05:27 736658 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/httpclient-4.5.2.jar
2016-10-31 09:05:09 327373 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/httpcore-4.4.5.jar
2016-10-31 09:05:21 564672 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/ion-java-1.0.1.jar
2016-10-31 09:05:24 39821 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-annotations-2.5.3.jar
2016-10-31 09:05:14 229998 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-core-2.5.3.jar
2016-10-31 09:05:16 1143162 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-databind-2.5.3.jar
2016-10-31 09:05:13 46519 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-dataformat-cbor-2.5.3.jar
2016-10-31 09:05:23 16617 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jcl-over-slf4j-1.7.10.jar
2016-10-31 09:05:32 26756 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jmespath-java-1.0.jar
2016-10-31 09:05:10 621992 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/joda-time-2.8.2.jar
2016-10-31 09:05:30 33031 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jsr305-3.0.0.jar
2016-10-31 09:05:30 489884 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/log4j-1.2.17.jar
2016-10-31 09:05:15 32119 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/slf4j-api-1.7.10.jar
2016-10-31 09:05:09 8866 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/slf4j-log4j12-1.7.10.jar

As I was able to retrieve the list, the S3 bucket seems operational.

We recommend you check for any proxy settings in your AMI that could be possibly interfering with the connection.

Adding the following environment variables may solve the problem.

OS environment variable: 
 * NO_PROXY="*.s3.dualstack.ap-southeast-2.amazonaws.com"

Java Property or JAVA_TOOL_OPTIONS: 
 * -Dhttp.nonProxyHosts="*.s3.dualstack.ap-southeast-2.amazonaws.com"

Other than that, if you have a valid support entitlement, we recommend you open a case with us so we can check if further:

https://support.atlassian.com/contact

Kind regards,

Eduardo Alvarenga
Atlassian Support APAC

All comments

Eduardo Alvarenga

Hello @Lyndon Howie

Thank you for reaching out to Atlassian Community!

Bamboo S3 buckets have an access list that would only accept Amazon's CIDRs, meaning that any non-AWS IP addresses will not be able to retrieve any data from the buckets.

I've tested it from an ordinary Elastic Agent and got the following:

root@ip-10-116-101-158:~# aws s3 ls --recursive s3://bamboo-agent-release-ap-se2/5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef
2016-10-31 09:05:23 17536 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/annotations-13.0.jar
2016-10-31 09:05:18 10399 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-annotations-1.1.0.jar
2016-10-31 09:05:21 37954 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-aws-bootstrap-1.0.132.jar
2016-10-31 09:05:32 69118 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-agent-bootstrap-5.14.0.2.jar
2016-10-31 09:05:17 30378 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-agent-elastic-5.14.0.2.jar
2016-10-31 09:05:34 7693 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-agent-elastic-shared-5.14.0.2.jar
2016-10-31 09:05:13 3754 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-api-agent-bootstrap-5.14.0.2.jar
2016-10-31 09:05:30 5077 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-bamboo-core-agent-bootstrap-5.14.0.2.jar
2016-10-31 09:05:10 49998 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/atlassian-tunnel-1.8.4.jar
2016-10-31 09:05:17 729956 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-core-1.11.40.jar
2016-10-31 09:05:09 3694062 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-ec2-1.11.40.jar
2016-10-31 09:05:15 427820 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-kms-1.11.40.jar
2016-10-31 09:05:10 655656 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/aws-java-sdk-s3-1.11.40-atlassian-1.jar
2016-10-31 09:05:14 3277268 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/bcprov-jdk15on-1.54.jar
2016-10-31 09:05:23 263865 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/commons-codec-1.8.jar
2016-10-31 09:05:16 60686 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/commons-logging-1.1.1.jar
2016-10-31 09:05:27 197860 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/gson-2.2.2-atlassian-1.jar
2016-10-31 09:05:27 736658 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/httpclient-4.5.2.jar
2016-10-31 09:05:09 327373 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/httpcore-4.4.5.jar
2016-10-31 09:05:21 564672 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/ion-java-1.0.1.jar
2016-10-31 09:05:24 39821 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-annotations-2.5.3.jar
2016-10-31 09:05:14 229998 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-core-2.5.3.jar
2016-10-31 09:05:16 1143162 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-databind-2.5.3.jar
2016-10-31 09:05:13 46519 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jackson-dataformat-cbor-2.5.3.jar
2016-10-31 09:05:23 16617 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jcl-over-slf4j-1.7.10.jar
2016-10-31 09:05:32 26756 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jmespath-java-1.0.jar
2016-10-31 09:05:10 621992 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/joda-time-2.8.2.jar
2016-10-31 09:05:30 33031 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/jsr305-3.0.0.jar
2016-10-31 09:05:30 489884 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/log4j-1.2.17.jar
2016-10-31 09:05:15 32119 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/slf4j-api-1.7.10.jar
2016-10-31 09:05:09 8866 5.14.0.2/1a074deaa55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef/boot/slf4j-log4j12-1.7.10.jar

As I was able to retrieve the list, the S3 bucket seems operational.

We recommend you check for any proxy settings in your AMI that could be possibly interfering with the connection.

Adding the following environment variables may solve the problem.

OS environment variable: 
 * NO_PROXY="*.s3.dualstack.ap-southeast-2.amazonaws.com"

Java Property or JAVA_TOOL_OPTIONS: 
 * -Dhttp.nonProxyHosts="*.s3.dualstack.ap-southeast-2.amazonaws.com"

Other than that, if you have a valid support entitlement, we recommend you open a case with us so we can check if further:

https://support.atlassian.com/contact

Kind regards,

Eduardo Alvarenga
Atlassian Support APAC

Lyndon Howie

Thanks for your response.

I checked our instance IP address (both in aws console and "whats my ip") and the instance's IP address of 3.25.88.68 appears to be in the published AWS CIDR range of 3.24.0.0/14

{
      "ip_prefix": "3.24.0.0/14",
      "region": "ap-southeast-2",
      "service": "AMAZON",
      "network_border_group": "ap-southeast-2"
    },

and we don't appear to have any proxy server configured.

I tried to also set the NO_PROXY env var as suggested but no luck.

C:\temp>aws s3 ls --recursive s3://bamboo-agent-release-ap-se2/5.14.0.2/1a074dea
a55086e27e844019f50d13829b411570a3d8f3e89807a61e5a5dcdef

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Acces
s Denied

Eduardo Alvarenga

Hey @Lyndon Howie,

Please try adding the --no-sign-request option to your aws s3 ls command.

It is likely that AWS is not allowing switching to anonymous when an invalid key is found anymore. I was able to reproduce your issue by modifying my credentials in .aws/credentials to an invalid one.

Maybe your AMI has some old Access key that is now invalid? Try checking your credentials file and environment variables.

Kind regards,

Eduardo Alvarenga
Atlassian Support APAC

Lyndon Howie

Hi Eduardo,

Thanks a lot for your responses, we did get it working in the end! Long story short, upgrading the version of the build agent (from 5.14 to 6.16) fixed the issue.

Long story long...

Adding the --no-sign-request did allow the aws s3 ls command to execute successfully. Adding s3:* permissions to our ec2 instance role also allowed aws s3 ls command to execute successfully (without --no-sign-request). However even with the s3:* permissions the agent was still getting the same error.

I then noticed that a different one of our build images that we rarely use was able to successfully start the agent and noticed it had a higher agent version. (Note that the newer agent didn't require the s3:* permissions to work, just upgrading the agent fixed the issue)