Out of space

[Mikkel Thomsen]

When building my artifacts, I tend to compress the files, and today I received the following error in my build:

gzip: stdout: No space left on device

So I checked my overall disk usage, and I use maybe 50/300 GB.

I proceeded with a "df" and the following was returned:

I must admit, my linux foo is not very strong, but to me, it seems like something is wrong, if a partition is 99% used, and the OS is complaining there is no more space.

So I went to the bamboo admin, and went to check how many builds and deployments were stored.

I decreased the numbers, deleting 6 builds, and 13 deployments.

I went to use df again, but this numbers here didn't change.

I tried google, but the only reasonable suggestion, was to change the partition size.

And this is the point where I'm no longer sure of what I'm doing, or, if this is even the issue.

Do you have any recommendations/tips for me? :) 

[Nic Brough -Adaptavist- · Accepted solution]

This is not really an Atlassian problem, just one of housekeeping a build server.

But it is one many of us run into a lot. 

I would recommend trying to work out where the problem is - find an answer to "what is eating all my disk space?"  The df command is excellent for telling you overall usage, but it has a far more powerful friend called du

Go to the root of the partition (in your case, it looks like a single disk, so "cd /") and as root, run "du --max-depth 1".  This will tell you which directories are biggest, so you can descend into each of them and repeat the command. 

My usual mistake is to not set up "log rotate" for /opt/atlassian-stuff, but another good one is having a system that errors a lot spitting out way too much data into /var/log (I've got a broken postfix on one of my home machines that's currently clobbering /var/log/mail*.log for about 2% of the disk every day...)

[Craig Castle-Mead]

Hi Mikkel,

One extra piece of information that may help explain your situation - Linux will often keep a  % of your partition available for root, this could explain why df is showing 99% used but the application is unable to write to it.

either way, running on 1% available is obviously risky!

Work through the suggestions from @Nic Brough -Adaptavist-  above to find where the space is going to and clean up.

CCM