Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,456,701
Community Members
 
Community Events
176
Community Groups

AWS Spot instance create failed with Invalid BASE64 encoding of user data

Hey, Atlassian Team

We starting to hit Spot Instance create failure with Invalid BASE64 encoding of user data since 2AM Nov 5th 2019 GMT time. 
We are using Bamboo 6.7.2 build 60706. 

We were hitting random issue like this before, maybe 2 out of 100 times, but we worked around it by store most of the userdata into S3 bucket. 

Since this morning, when we enable Spot Instance Support in Bamboo, we will start hit this error 100% time. This feels like some AWS side change. Do you have any suggestion? 

 

We noticed this JIRA https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

 

This JIRA is saying deliver a fix in Bamboo 6.9. https://jira.atlassian.com/browse/BAM-19945?focusedCommentId=1944640&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

 

Will update to 6.9 gives back the ability to use Spot Agent for bamboo? 

 

 

 

 

Errors logs

System Error Details
Elastic Bamboo Error : EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXX failed.
(com.amazonaws.services.ec2.model.AmazonEC2Exception : Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1))

Occurred: 05 Nov 2019, 6:54:33 AM
com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: ab6ad633-d10d-4a79-a725-e9e5c2049af1)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:17597)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17566)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17555)
at com.amazonaws.services.ec2.AmazonEC2Client.executeRequestSpotInstances(AmazonEC2Client.java:16281)
at com.amazonaws.services.ec2.AmazonEC2Client.requestSpotInstances(AmazonEC2Client.java:16252)
at sun.reflect.GeneratedMethodAccessor3136.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.aws.CallTimingProxy$1.invoke(CallTimingProxy.java:34)
at com.sun.proxy.$Proxy1639.requestSpotInstances(Unknown Source)
at com.atlassian.aws.ec2.awssdk.launch.AwsSpotInstanceLauncher.call(AwsSpotInstanceLauncher.java:77)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.launchInstance(RemoteEC2InstanceImpl.java:363)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.backgroundStart(RemoteEC2InstanceImpl.java:346)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl.access$100(RemoteEC2InstanceImpl.java:39)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$1.run(RemoteEC2InstanceImpl.java:95)
at com.atlassian.aws.ec2.RemoteEC2InstanceImpl$CatchingRunnableDecorator.run(RemoteEC2InstanceImpl.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)




Nov 5, 2019 6:54:33 AM Requested that new elastic instance be created for configuration: Quick Test / ami-XXXXXXX
Nov 5, 2019 6:54:33 AM Detected that a pending instance request for image [Quick Test], ami: ami-XXXXXXX has been abandoned.

 

 

 

1 answer

1 accepted

2 votes
Answer accepted

Hi @Lei P

I saw you opened a ticket with our support team for this issue and it was already resolved.
I'm adding the findings here in case other users face the same issue.

Problem

This problem may happen eventually. The issue is caused by the following bug which was fixed in Bamboo 6.9.0:

Resolution

Upgrade the Bamboo instance to 6.9.0 or any newer version.

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events