How do I debug "no space left on device" in Bamboo OnDemand?

[Ben]

I'm getting the same error quite often in various project builds in Bamboo OnDemand - it won't clone my repos. I have a number of projects, each with their own repo, but I can't imagine I'm actually filling up the space on the EC2 instance...

Job error summary:

The build has failed to check out the source code: : No space left on device

From the "full build log"

23-Jan-2014 21:00:42	Build libjava_store - develop - Default Job #14 (STORE-DEV-JOB1-14) started building on agent Elastic Agent on i-c9a0e0e9
23-Jan-2014 21:00:42	Build working directory is /mnt/bamboo-ebs/bamboo-agent/build-dir/STORE-DEV-JOB1
23-Jan-2014 21:00:42	Error occurred while executing the build for libjava_store - develop - Default Job #14 (STORE-DEV-JOB1-14) : : No space left on device

I'm running on an Amazon Linux EBS x86_64, M1 LARGE, which should have:

7.5 GiB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage

I only do shallow clones, and not the entire repo (we use bitbucket) for the repositories. And even if I did get the whole repo and all history, my repos just aren't that big. So I can't figure out what's up.

I'm sure I've configured something wrong, but I have no idea how to figure out what...

Any ideas are welcome!

[Ben · Accepted solution]

We were unaware that the default bamboo-ebs image moved the default build-dir from the /media/ephemeral0/home (which seems to be ~400GB) to use the /mnt/bamboo-ebs (which for us is ony 1GB, since we use it just for settings.xml and to bootstrap the maven repo with all the basics).

Is there a rational as to why this is the default behaviour, when creating an EBS?

We are trying just to remove bamboo-agent/bamboo-agent.cfg.xml as per the documentation, which will hopefully resolve the problem. It seems to me that the bamboo documentation around all of this is pretty poor. We had a hard time figuring out how to share settings.xml in the first place (which seems like it would be a pretty much required step for all maven users).

[Malte Hübner]

I think this is exactly what you might be searching for: "Use EC2 ephemeral storage as build directory even with an EBS volume attached": https://confluence.atlassian.com/display/BAMKB/Use+EC2+ephemeral+storage+as+build+directory+even+with+an+EBS+volume+attached

[Shai]

I am having the same issue.

The script location describe in the article is wrong. i can't find it on my ec2 instance.

Any idea?

10x

[Przemek Bruski]

It looiks like you've customised your instance and set your build directory to your EBS volume - so, you're limited by the EBS size. The default build directory is on ephemeral storage ans has hundreds of GBs available.

You have to either increase the EBS size or stay with the default build directory location.

[Ben]

We're running OnDemand, and I don't recall changing any such configuration.

In Bamboo Admin - System Information, it shows:

Current running directory/sw/java/tomcat/apache-tomcat-6.0.32

Configuration path/data/jirastudio/bamboo/home/xml-data/configuration

Build path/data/jirastudio/bamboo/home/xml-data/builds

Build working directory/data/jirastudio/bamboo/home/xml-data/build-dir

Artifacts directory/data/jirastudio/bamboo/home/artifacts

Bamboo home/data/jirastudio/bamboo/home

Bamboo logs/data/jirastudio/bamboo/home/logs/atlassian-bamboo.log

Temporary directory/data/service/j2ee_bamboo/catalina-base/temp

User home /data/jirastudio/bamboo/home

which does not seem to indicate the /mnt/bamboo-ebs/

path listed above in the errors.

[Przemek Bruski]

Your build takes place on the build agent. All paths you've listed are from the server.

[Ben]

Ok, that makes sense - so how do I configure the build directory on elastic bamboo agents for OnDemand? I don't seen anything for that, but I'm sure I'm just not looking in the right place.

[Ben]

Digging further into the docs, I finally found this:

• bamboo-agent/bamboo-agent.cfg.xml- This configuration file modifies the build working directory to point to build working directory on the EBS volume.

I didn't modify this, afaik. So I think you're tell me that the default behaviour in Bamboo when using an attached EBS (which we use to share maven settings.xml, etc) is to use the EBS as the working directory. So I gues all I have to do is modify or remove this file and create a a new EBS snapshot, and the problem should go away.