AWS Spot instance create failed with Invalid BASE64 encoding of user data

[Lei P]

Hey, Atlassian Team

We starting to hit Spot Instance create failure with Invalid BASE64 encoding of user data since 2AM Nov 5th 2019 GMT time. 
We are using Bamboo 6.7.2 build 60706. 

We were hitting random issue like this before, maybe 2 out of 100 times, but we worked around it by store most of the userdata into S3 bucket. 

Since this morning, when we enable Spot Instance Support in Bamboo, we will start hit this error 100% time. This feels like some AWS side change. Do you have any suggestion? 

We noticed this JIRA https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

This JIRA is saying deliver a fix in Bamboo 6.9. https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

Will update to 6.9 gives back the ability to use Spot Agent for bamboo? 

Errors logs

System Error Details
Elastic Bamboo Error : EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXX failed.
(com.amazonaws.services.ec2.model.AmazonEC2Exception : Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1))

Occurred: 05 Nov 2019, 6:54:33 AM
com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:17597)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17566)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17555)
at com.amazonaws.services.ec2.AmazonEC2Client.executeRequestSpotInstances(AmazonEC2Client.java:16281)
at com.amazonaws.services.ec2.AmazonEC2Client.requestSpotInstances(AmazonEC2Client.java:16252)
at sun.reflect.GeneratedMethodAccessor3136.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.aws.CallTimingProxy$1.invoke(CallTimingProxy.java:34)
at com.sun.proxy.$Proxy1639.requestSpotInstances(Unknown Source)
at com.atlassian.aws.ec2.awssdk.launch.AwsSpotInstanceLauncher.call(AwsSpotInstanceLauncher.java:77)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.launchInstance(RemoteEC2InstanceImpl.java:363)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.backgroundStart(RemoteEC2InstanceImpl.java:346)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.access$100(RemoteEC2InstanceImpl.java:39)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$1.run(RemoteEC2InstanceImpl.java:95)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$CatchingRunnableDecorator.run(RemoteEC2InstanceImpl.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)




Nov 5, 2019 6:54:33 AM Requested that new elastic instance be created for configuration: Quick Test / ami-XXXXXXX
Nov 5, 2019 6:54:33 AM Detected that a pending instance request for image [Quick Test], ami: ami-XXXXXXX has been abandoned.

 

 

[Daniel Santos · Accepted solution]

Hi @Lei P

I saw you opened a ticket with our support team for this issue and it was already resolved.
I'm adding the findings here in case other users face the same issue.

Problem

This problem may happen eventually. The issue is caused by the following bug which was fixed in Bamboo 6.9.0:

•  [BAM-19945] Elastic Bamboo - Invalid Base64 encoding of user data

Resolution

Upgrade the Bamboo instance to 6.9.0 or any newer version.