Hey, Atlassian Team
We starting to hit Spot Instance create failure with Invalid BASE64 encoding of user data since 2AM Nov 5th 2019 GMT time.
We are using Bamboo 6.7.2 build 60706.
We were hitting random issue like this before, maybe 2 out of 100 times, but we worked around it by store most of the userdata into S3 bucket.
Since this morning, when we enable Spot Instance Support in Bamboo, we will start hit this error 100% time. This feels like some AWS side change. Do you have any suggestion?
We noticed this JIRA https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
This JIRA is saying deliver a fix in Bamboo 6.9. https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
Will update to 6.9 gives back the ability to use Spot Agent for bamboo?
Errors logs
System Error Details
Elastic Bamboo Error : EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXX failed.
(com.amazonaws.services.ec2.model.AmazonEC2Exception : Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1))
Occurred: 05 Nov 2019, 6:54:33 AM
com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:17597)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17566)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17555)
at com.amazonaws.services.ec2.AmazonEC2Client.executeRequestSpotInstances(AmazonEC2Client.java:16281)
at com.amazonaws.services.ec2.AmazonEC2Client.requestSpotInstances(AmazonEC2Client.java:16252)
at sun.reflect.GeneratedMethodAccessor3136.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.aws.CallTimingProxy$1.invoke(CallTimingProxy.java:34)
at com.sun.proxy.$Proxy1639.requestSpotInstances(Unknown Source)
at com.atlassian.aws.ec2.awssdk.launch.AwsSpotInstanceLauncher.call(AwsSpotInstanceLauncher.java:77)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.launchInstance(RemoteEC2InstanceImpl.java:363)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.backgroundStart(RemoteEC2InstanceImpl.java:346)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.access$100(RemoteEC2InstanceImpl.java:39)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$1.run(RemoteEC2InstanceImpl.java:95)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$CatchingRunnableDecorator.run(RemoteEC2InstanceImpl.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Nov 5, 2019 6:54:33 AM Requested that new elastic instance be created for configuration: Quick Test / ami-XXXXXXX
Nov 5, 2019 6:54:33 AM Detected that a pending instance request for image [Quick Test], ami: ami-XXXXXXX has been abandoned.
Hi @Lei P
I saw you opened a ticket with our support team for this issue and it was already resolved.
I'm adding the findings here in case other users face the same issue.
Problem
This problem may happen eventually. The issue is caused by the following bug which was fixed in Bamboo 6.9.0:
Resolution
Upgrade the Bamboo instance to 6.9.0 or any newer version.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.