Instance Startup Script vs EC2 User Data on Windows Elastic Agents

[nikshornikov]

I checked out this question: https://answers.atlassian.com/questions/157864/elastic-agent-ec2-user-data-vs-instance-startup-script but the OP trailed off, and the answer was not at all detailed.

Running a (very small) risk of this question becoming obsolete tomorrow, I'm referring to Elastic Agents off the ami-976edafe AMI.

Can anyone explain which component on the instance will pick up the script, where it will pick it up from, and what it'll execute it through? My first attempt was passing it data as I would to the agent (cmd/bat-like), but I found no evidence anywhere on the agent instance that anything ran. Should I be passing script or powershell tags (as I would for a modern AWS base Windows AMI)?

Would the answer be any different if I used my own AMI based on today's AWS images (where ec2config would indeed work with script and powershell tags)?

Thank you in advance.

[nikshornikov · Accepted solution]

For Windows, the difference is that EC2 User Data is not something you can pass to an Elastic Agent (or any remote agent with On Demand), while the Instance Startup Script is something which the agent, as of today, will fail to parse (there is no documentation on syntax constraints). Effectively they are the same.

[Michael Wheelaghan]

Is the guide to setup a custom windows ami a relatively new process? Months ago I had a custom windows ami doing elastic builds and running plans without much effort , ie create image and add it to Image Configurations. After trying several different windows ami's It looks as though there are no c:\opt folders created when I login to the instance, and Remote Agent status is always 'Pending'. Do you need to manually setup user account autologon and setup elastic agent jar?

[Przemek Bruski]

The script gets passed on to Windows instances just like Unix instances. But there's nothing that will run it out of the box. This only happens on Unix instances.

[Przemek Bruski]

EBSes on Windows will be supported with Bamboo 5.0

[hughperkins]

Just to double-check I'm not misunderstanding, in the 'Edit Elastic Image Configuration', for Windows images, everything in the 'Instance Startup Script' is effectively completely ignored? What's the easiest way of working around this? I don't really want to create a custom image, since then each time you update Bamboo, I'll need to create a new image. But let's suppose that's the only possible way, how would I do that?

[Przemek Bruski]

Yes, it's ignored. Custom images get updated automatically, you don't need to do it when you upgrade. You can find the instructions here: https://confluence.atlassian.com/display/BAMBOO/Creating+a+custom+elastic+image+-+Windows .

[hughperkins]

Ok, I've started a bamboo windows-64 image, created an account called 'bootstrap', created a scheduled task, which runs on network startup, as bootstrap, downloads a script from my webserver, and runs it. Then, I created an AMI from this instance, using the aws console, and started a new instance from the bamboo console, from this new ami. However, the scheduled task still has the old computer name in the runas credentials, so the task doesn't run. The scheduled task created by bamboo runs just fine. How did you fix this issue for the bamboo task, so that after computer rename, the task still runs ok?

[nikshornikov]

Did you set up that task and user the same way? (run w/e or not logged in and log on as batch job) Also, I did not sysprep or shut down before snapshotting, but that might not be relevant if Atlassian's prep script also wipes the computername.

PS. "Ignored" isn't exactly what happens to the startup script. It'll still get read and potentially crash the agent.

[hughperkins]

@Nik, oops I didn't see you'd answered my comment, before I deleted my comment. It turns out that either the task was running ok all the time, or deselecting 'Do not store password' fixed the issue with my task scheduler, that didn't seem to be running for me, after the ecconfig computer rename.

[Philip Hardy]

Is this still the case with Bamboo 5.4-OD-3?

Thanks in advance.

[Przemek Bruski]

Yes, it is, sorry.

[FrontlineRecruitment IT]

It seems like this was supposed to be fixed (https://jira.atlassian.com/browse/BAM-10089). Yet, I'm still experiencing problems mounting the disks with my OnDemand instance. Is this in fact not fixed?

[Haroon Said]

If someone wants to 'how do I install' using instance start-up on Bamboo Agent, you can use the following example:

@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
choco feature enable -n allowGlobalConfirmation
choco install python -y
choco install awscli -y
choco install nuget.commandline -y
choco install visualstudio2019professional -y
choco install visualstudio2019-workload-netweb -y
choco install visualstudio2019-workload-netcoretools -y
choco install visualstudio2019-workload-manageddesktop -y
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command

[nikshornikov]

Pretty sad that people are still landing here. This sounds like the closest thing to a workaround!

Someone might also find it useful that powershell.exe has an EncodedCommand switch (That's how bad the double-quote thing is), and surprising to know that certutil can base64 encode-decode in batch scripts.

[Philip Hardy]

This is probably too late to be of use to you, but it may help others in finding solutions.

If you base64 encode your script and enter it into the image configuration page it will not interfere with the agent's parsing.

Then on the instance you can have something like the following run to parse the script from the user data, decode, and execute.

$json = (new-object System.Net.WebClient).DownloadString("http://169.254.169.254/latest/user-data")

$userData = $json | ConvertFrom-Json

$script = [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String($userData.metaData.startupScript_0))

$script -split "`r`n" | ForEach-Object { cmd /c $_ }

Of course there's scope here for differentiating parsing of powershell vs batch and executing appropriately amongst other things.

Hope this helps someone.

Phil

[nikshornikov]

I tried passing a startup script without the tags, but still got the same failure in agent.out. Running with no script specified lets the agent boot up. I do hate being right, but this is a spectacular troll indeed -- I laughed for two minutes. Does anyone have any idea of what it would take to get a script to the agent on windows as far as syntax? Scripts have a way of containing quotes and slashes and weird characters like that.

Exception in thread "main" com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: invalid number or unquoted string near ?8<?b\ab\ab\ab\ab\ab\ab  
	at com.google.gson.Streams.parse(Streams.java:51)
	at com.google.gson.Gson.fromJson(Gson.java:514)
	at com.google.gson.Gson.fromJson(Gson.java:484)
	at com.google.gson.Gson.fromJson(Gson.java:434)
	at com.google.gson.Gson.fromJson(Gson.java:406)
	at com.atlassian.aws.ec2.EC2Utils.getUserData(EC2Utils.java:117)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.<init>(ElasticAgentInstaller.java:41)
	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:61)
Caused by: com.google.gson.stream.MalformedJsonException: invalid number or unquoted string near ?8<?b\ab\ab\ab\ab\ab\ab  
	at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1110)
	at com.google.gson.stream.JsonReader.decodeLiteral(JsonReader.java:1100)
	at com.google.gson.stream.JsonReader.peek(JsonReader.java:343)
	at com.google.gson.Streams.parse(Streams.java:38)
	... 7 more

[nikshornikov]

I see where the script got buried in user data for the instance and still see no evidence of it having been attempted by anything (of course, agent installation failed for some reason anyway so here goes another 20 minute exercise). It does seem less likely that I'm supposed to format it like user data. Currently my money's on Bamboo On Demand just trolling the Windows crowd with this text box (just like EBS volume attachment).