Hi, I'm trying to create custom build server instances from Bamboo elastic images. I'm installing .NET 4.5 and a Visual Studio Shell on the drive. But the elastic images are terminating suddenly. I'll be logged on and moving forward with building the build server, and it just shuts down and terminates. Please see the below log from Bamboo. Why are the Bamboo elastic images terminating unexpectedly??
Oct 12, 2012 1:53:10 PM Requested that new elastic instance be created for configuration: Windows .NET 4.5 Build Server |
Oct 12, 2012 1:53:10 PM Elastic instance [i-80f9dbfd] transitioned from STARTING to IDENTIFIED. |
Oct 12, 2012 1:53:52 PM Elastic instance [i-80f9dbfd] transitioned from IDENTIFIED to RUNNING. |
Oct 12, 2012 2:02:35 PM An elastic agent is loading on instance: [i-80f9dbfd] |
Oct 12, 2012 2:04:01 PM Elastic Agent "Elastic Agent on i-80f9dbfd" started on instance i-80f9dbfd |
Oct 12, 2012 3:04:12 PM An elastic agent is loading on instance: [i-80f9dbfd] |
Oct 12, 2012 3:09:17 PM Elastic Agent "Elastic Agent on i-80f9dbfd" stopped on instance i-80f9dbfd |
Oct 12, 2012 3:09:51 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" started on instance i-80f9dbfd |
Oct 12, 2012 3:14:36 PM Elastic instance [i-80f9dbfd] transitioned from RUNNING to SHUTTING_DOWN. |
Oct 12, 2012 3:14:57 PM Elastic instance [i-80f9dbfd] transitioned from SHUTTING_DOWN to TERMINATED. |
Oct 12, 2012 3:14:57 PM Detected that the elastic instance [i-80f9dbfd] has been terminated. |
Oct 12, 2012 3:14:57 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" stopped on instance i-80f9dbfd |
I've spoken with Atlassian technical support and the problem is caused by the automatic elastic instance management configuration setting. It is the best practice to disable this feature. There is a Bamboo agent that runs on each Bamboo build server instance. If this agent stops and the automatic elastic instance management configuration setting is enabled, then the server auto-terminates. The agent may stop if you restart the Bamboo build server instance.
I have disabled everything that could possibly auto-terminate my elastic images, but they are still terminating on their own (like once a month - fairly infrequently but still annoying). Any other ideas to ensure that the image doesn't shut down? Is Atlasssian shutting down their EC2 cloud group or something like that periodically?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi,
I didn’t see a specific answer here, it’s an old thread so, I suspect it was resolved. We recently encountered the same symptoms, just leaving my experience for future visitors.
Bamboo would launch elastic instance, the instance would terminate about 3-4 minutes later, the agent never moved out of pending.
Cause:
Our AMI was setup to create 2 volumes when new instances are launched.
When the instance was terminated, the second volume was being left behind. I suspect this has something to do with our AMI profile permissions. Eventually, AWS wouldn’t provision anymore volumes because, we had too many created. (got up to around 100 orphaned volumes before we noticed)
Result:
Bamboo would attempt to create new elastic instances
AWS would start the instance, then terminate it because the volume couldn’t be created
Inside Bamboo it appeared the agent was just randomly going offline
For us, we just adjust the AMI to only create 1 volume and used the asw console to cleanup the volumes.
cheers
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi ,
I am also observing similar error. Bamboo elastic agents get terminated while building plans.
any workaround and solution?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I suspect that the agent is having problems starting up, shuts down, gets restarted (that why you see the agent marked with (2) ). Eventually Bamboo decides the agent won't come up and kills the instance.
Are you stopping/starting/restarting the instance in the meantime?
If not, what does the agent log say (you can find it in the agent home directory)?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thanks for your response! Please see the attached Elastic Agent Usage History. Each of the Windows Bamboo build server instances have auto-terminated. I created the three most recent elastic agents as a test. I never remoted into these three instances, but they each terminated at 7:48 PM on Oct 14. I do not stop/start the instances after their initial creation. I did restart some of these build servers after I installed .NET 4.5. We use AWS on a daily basis and we only have this problem with the Bamboo elastic agents. This is a serious impediment to our progress. Thanks so much for your time & consideration.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Elastic Agent History › Elastic Agent on i-26f4d95b (elastic)
Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server
You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.
Agent Details
0%0 / 0 Successful
Elastic instance Instance i-26f4d95b (Windows .NET 4.5 Build Server image configuration)
The elastic instance this agent is/was running on.
Last startup timeOct 12, 2012 4:06:53 PM
Last shutdown timeOct 14, 2012 7:48:07 PM
Up time2 days, 3 hours, 41 minutes
Build History
The agent was building for 0% of the up time.
There are no builds to display.
Elastic Agent History › Elastic Agent on i-80f9dbfd (elastic)
Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server
You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.
Agent Details
0%0 / 0 Successful
Elastic instance Instance i-80f9dbfd (Windows .NET 4.5 Build Server image configuration)
The elastic instance this agent is/was running on.
Last startup timeOct 12, 2012 2:04:01 PM
Last shutdown timeOct 12, 2012 3:09:17 PM
Up time1 hour, 5 minutes
Build History
The agent was building for 0% of the up time.
There are no builds to display.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Elastic Agent Usage History
Listed below are all elastic agents that have run in the past. This does not include any currently running elastic agents, you can view these on the Elastic Instances page
Agent |
Configuration |
Start Time |
Shutdown Time |
Uptime |
Successful Jobs |
Operations |
12 Oct 2012, 4:09:16 PM |
14 Oct 2012, 7:48:07 PM |
2 days, 3 hours, 38 minutes |
No builds |
|||
12 Oct 2012, 4:06:53 PM |
14 Oct 2012, 7:48:07 PM |
2 days, 3 hours, 41 minutes |
No builds |
|||
12 Oct 2012, 4:06:39 PM |
14 Oct 2012, 7:48:07 PM |
2 days, 3 hours, 41 minutes |
No builds |
|||
12 Oct 2012, 3:09:51 PM |
12 Oct 2012, 3:14:57 PM |
5 minutes |
0 / 1 |
|||
12 Oct 2012, 2:04:01 PM |
12 Oct 2012, 3:09:17 PM |
1 hour, 5 minutes |
No builds |
|||
12 Oct 2012, 1:31:58 PM |
12 Oct 2012, 1:45:48 PM |
13 minutes |
0 / 1 |
|||
12 Oct 2012, 1:12:49 PM |
12 Oct 2012, 1:40:17 PM |
27 minutes |
0 / 1 |
|||
12 Oct 2012, 12:38:13 PM |
12 Oct 2012, 12:51:46 PM |
13 minutes |
No builds |
|||
10 Oct 2012, 11:17:28 AM |
12 Oct 2012, 12:46:17 PM |
2 days, 1 hour, 28 minutes |
8 / 28 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Elastic Agent History › Elastic Agent on i-26f4d95b (elastic)
Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server
You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.
Agent Details
0%0 / 0 Successful
Elastic instance Instance i-26f4d95b (Windows .NET 4.5 Build Server image configuration)
The elastic instance this agent is/was running on.
Last startup timeOct 12, 2012 4:06:53 PM
Last shutdown timeOct 14, 2012 7:48:07 PM
Up time2 days, 3 hours, 41 minutes
Build History
The agent was building for 0% of the up time.
There are no builds to display.
Elastic Agent History › Elastic Agent on i-80f9dbfd (elastic)
Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server
You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.
Agent Details
0%0 / 0 Successful
Elastic instance Instance i-80f9dbfd (Windows .NET 4.5 Build Server image configuration)
The elastic instance this agent is/was running on.
Last startup timeOct 12, 2012 2:04:01 PM
Last shutdown timeOct 12, 2012 3:09:17 PM
Up time1 hour, 5 minutes
Build History
The agent was building for 0% of the up time.
There are no builds to display.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Do you have Elastic Instance Management enabled in your Elastic Bamboo configuration?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?
Thanks for your reply!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Elastic Bamboo is configured with our Amazon Web Services Account. Automatic Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?
Thanks for your reply!
-----------------------------------------------------------------------------------------
Support for spot instances is disabled.
AWS account identifier files are not automatically uploaded to new elastic instances.
EC2 instances must be manually created and terminated as needed.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.