AWS Spot instance create failed with Invalid BASE64 encoding of user data

Lei P November 5, 2019

Hey, Atlassian Team

We starting to hit Spot Instance create failure with Invalid BASE64 encoding of user data since 2AM Nov 5th 2019 GMT time. 
We are using Bamboo 6.7.2 build 60706. 

We were hitting random issue like this before, maybe 2 out of 100 times, but we worked around it by store most of the userdata into S3 bucket. 

Since this morning, when we enable Spot Instance Support in Bamboo, we will start hit this error 100% time. This feels like some AWS side change. Do you have any suggestion? 

 

We noticed this JIRA https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

 

This JIRA is saying deliver a fix in Bamboo 6.9. https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

 

Will update to 6.9 gives back the ability to use Spot Agent for bamboo? 

 

 

 

 

Errors logs

System Error Details
Elastic Bamboo Error : EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXX failed.
(com.amazonaws.services.ec2.model.AmazonEC2Exception : Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1))

Occurred: 05 Nov 2019, 6:54:33 AM
com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:17597)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17566)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17555)
at com.amazonaws.services.ec2.AmazonEC2Client.executeRequestSpotInstances(AmazonEC2Client.java:16281)
at com.amazonaws.services.ec2.AmazonEC2Client.requestSpotInstances(AmazonEC2Client.java:16252)
at sun.reflect.GeneratedMethodAccessor3136.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.aws.CallTimingProxy$1.invoke(CallTimingProxy.java:34)
at com.sun.proxy.$Proxy1639.requestSpotInstances(Unknown Source)
at com.atlassian.aws.ec2.awssdk.launch.AwsSpotInstanceLauncher.call(AwsSpotInstanceLauncher.java:77)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.launchInstance(RemoteEC2InstanceImpl.java:363)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.backgroundStart(RemoteEC2InstanceImpl.java:346)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.access$100(RemoteEC2InstanceImpl.java:39)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$1.run(RemoteEC2InstanceImpl.java:95)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$CatchingRunnableDecorator.run(RemoteEC2InstanceImpl.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)




Nov 5, 2019 6:54:33 AM Requested that new elastic instance be created for configuration: Quick Test / ami-XXXXXXX
Nov 5, 2019 6:54:33 AM Detected that a pending instance request for image [Quick Test], ami: ami-XXXXXXX has been abandoned.

 

 

 

1 answer

1 accepted

2 votes
Answer accepted
Daniel Santos
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
November 6, 2019

Hi @Lei P

I saw you opened a ticket with our support team for this issue and it was already resolved.
I'm adding the findings here in case other users face the same issue.

Problem

This problem may happen eventually. The issue is caused by the following bug which was fixed in Bamboo 6.9.0:

Resolution

Upgrade the Bamboo instance to 6.9.0 or any newer version.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events