Elastic Images Terminate

[Galen Pyle]

Hi, I'm trying to create custom build server instances from Bamboo elastic images. I'm installing .NET 4.5 and a Visual Studio Shell on the drive. But the elastic images are terminating suddenly. I'll be logged on and moving forward with building the build server, and it just shuts down and terminates. Please see the below log from Bamboo. Why are the Bamboo elastic images terminating unexpectedly??

Oct 12, 2012 1:53:10 PM Requested that new elastic instance be created for configuration: Windows .NET 4.5 Build Server

Oct 12, 2012 1:53:10 PM Elastic instance [i-80f9dbfd] transitioned from STARTING to IDENTIFIED.

Oct 12, 2012 1:53:52 PM Elastic instance [i-80f9dbfd] transitioned from IDENTIFIED to RUNNING.

Oct 12, 2012 2:02:35 PM An elastic agent is loading on instance: [i-80f9dbfd]

Oct 12, 2012 2:04:01 PM Elastic Agent "Elastic Agent on i-80f9dbfd" started on instance i-80f9dbfd

Oct 12, 2012 3:04:12 PM An elastic agent is loading on instance: [i-80f9dbfd]

Oct 12, 2012 3:09:17 PM Elastic Agent "Elastic Agent on i-80f9dbfd" stopped on instance i-80f9dbfd

Oct 12, 2012 3:09:51 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" started on instance i-80f9dbfd

Oct 12, 2012 3:14:36 PM Elastic instance [i-80f9dbfd] transitioned from RUNNING to SHUTTING_DOWN.

Oct 12, 2012 3:14:57 PM Elastic instance [i-80f9dbfd] transitioned from SHUTTING_DOWN to TERMINATED.

Oct 12, 2012 3:14:57 PM Detected that the elastic instance [i-80f9dbfd] has been terminated.

Oct 12, 2012 3:14:57 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" stopped on instance i-80f9dbfd

[Galen Pyle · Accepted solution]

I've spoken with Atlassian technical support and the problem is caused by the automatic elastic instance management configuration setting. It is the best practice to disable this feature. There is a Bamboo agent that runs on each Bamboo build server instance. If this agent stops and the automatic elastic instance management configuration setting is enabled, then the server auto-terminates. The agent may stop if you restart the Bamboo build server instance.

[Jeff Hansen]

I have disabled everything that could possibly auto-terminate my elastic images, but they are still terminating on their own (like once a month - fairly infrequently but still annoying). Any other ideas to ensure that the image doesn't shut down? Is Atlasssian shutting down their EC2 cloud group or something like that periodically?

[derek williams]

Hi,

I didn’t see a specific answer here, it’s an old thread so, I suspect it was resolved.  We recently encountered the same symptoms, just leaving my experience for future visitors.

Bamboo would launch elastic instance, the instance would terminate about 3-4 minutes later, the agent never moved out of pending.

Cause:

• Our AMI was setup to create 2 volumes when new instances are launched.  

• When the instance was terminated, the second volume was being left behind.  I suspect this has something to do with our AMI profile permissions.  Eventually, AWS wouldn’t provision anymore volumes because, we had too many created.  (got up to around 100 orphaned volumes before we noticed)

Result:

• Bamboo would attempt to create new elastic instances

• AWS would start the instance, then terminate it because the volume couldn’t be created

• Inside Bamboo it appeared the agent was just randomly going offline

For us, we just adjust the AMI to only create 1 volume and used the asw console to cleanup the volumes.

cheers

[140401 kindia]

Hi ,

I am also observing similar error. Bamboo elastic agents get terminated while building plans.

any workaround and solution?

[Przemek Bruski]

I suspect that the agent is having problems starting up, shuts down, gets restarted (that why you see the agent marked with (2) ). Eventually Bamboo decides the agent won't come up and kills the instance.

Are you stopping/starting/restarting the instance in the meantime?

If not, what does the agent log say (you can find it in the agent home directory)?

[Galen Pyle]

Thanks for your response! Please see the attached Elastic Agent Usage History. Each of the Windows Bamboo build server instances have auto-terminated. I created the three most recent elastic agents as a test. I never remoted into these three instances, but they each terminated at 7:48 PM on Oct 14. I do not stop/start the instances after their initial creation. I did restart some of these build servers after I installed .NET 4.5. We use AWS on a daily basis and we only have this problem with the Bamboo elastic agents. This is a serious impediment to our progress. Thanks so much for your time & consideration.

[Galen Pyle]

Elastic Agent History › Elastic Agent on i-26f4d95b (elastic)

Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-26f4d95b (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 4:06:53 PM

Last shutdown timeOct 14, 2012 7:48:07 PM

Up time2 days, 3 hours, 41 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

Elastic Agent History › Elastic Agent on i-80f9dbfd (elastic)

Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-80f9dbfd (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 2:04:01 PM

Last shutdown timeOct 12, 2012 3:09:17 PM

Up time1 hour, 5 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

[Galen Pyle]

Elastic Agent Usage History

Listed below are all elastic agents that have run in the past. This does not include any currently running elastic agents, you can view these on the Elastic Instances page

Agent	Configuration	Start Time	Shutdown Time	Uptime	Successful Jobs	Operations
Elastic Agent on i-24f4d959	Windows .NET 4.5 Build Server	12 Oct 2012, 4:09:16 PM	14 Oct 2012, 7:48:07 PM	2 days, 3 hours, 38 minutes	No builds	View | Delete
Elastic Agent on i-26f4d95b	Windows .NET 4.5 Build Server	12 Oct 2012, 4:06:53 PM	14 Oct 2012, 7:48:07 PM	2 days, 3 hours, 41 minutes	No builds	View | Delete
Elastic Agent on i-42f4d93f	Windows .NET 4.5 Build Server	12 Oct 2012, 4:06:39 PM	14 Oct 2012, 7:48:07 PM	2 days, 3 hours, 41 minutes	No builds	View | Delete
Elastic Agent on i-80f9dbfd (2)	Windows .NET 4.5 Build Server	12 Oct 2012, 3:09:51 PM	12 Oct 2012, 3:14:57 PM	5 minutes	0 / 1	View | Delete
Elastic Agent on i-80f9dbfd	Windows .NET 4.5 Build Server	12 Oct 2012, 2:04:01 PM	12 Oct 2012, 3:09:17 PM	1 hour, 5 minutes	No builds	View | Delete
Elastic Agent on i-84496bf9 (2)	Windows .NET 4.5 Build Server	12 Oct 2012, 1:31:58 PM	12 Oct 2012, 1:45:48 PM	13 minutes	0 / 1	View | Delete
Elastic Agent on i-84496bf9	Windows .NET 4.5 Build Server	12 Oct 2012, 1:12:49 PM	12 Oct 2012, 1:40:17 PM	27 minutes	0 / 1	View | Delete
Elastic Agent on i-aa652fd7	Default Image EBS x86_64 (windows)	12 Oct 2012, 12:38:13 PM	12 Oct 2012, 12:51:46 PM	13 minutes	No builds	View | Delete
Windows .NET 4.5 Build Server	Default Image EBS x86_64 (windows)	10 Oct 2012, 11:17:28 AM	12 Oct 2012, 12:46:17 PM	2 days, 1 hour, 28 minutes	8 / 28	View | Delete

[Galen Pyle]

Elastic Agent History › Elastic Agent on i-26f4d95b (elastic)

Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-26f4d95b (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 4:06:53 PM

Last shutdown timeOct 14, 2012 7:48:07 PM

Up time2 days, 3 hours, 41 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

Elastic Agent History › Elastic Agent on i-80f9dbfd (elastic)

Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-80f9dbfd (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 2:04:01 PM

Last shutdown timeOct 12, 2012 3:09:17 PM

Up time1 hour, 5 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

[Przemek Bruski]

Do you have Elastic Instance Management enabled in your Elastic Bamboo configuration?

[Galen Pyle]

Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?

Thanks for your reply!

[Galen Pyle]

Elastic Bamboo is configured with our Amazon Web Services Account. Automatic Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?

Thanks for your reply!

-----------------------------------------------------------------------------------------

Spot Instances Configuration

Support for spot instances is disabled.

Elastic Bamboo AWS Settings

AWS account identifier files are not automatically uploaded to new elastic instances.

Automatic Elastic Instance Management

EC2 instances must be manually created and terminated as needed.