Does Bamboo sometimes remove current working directory?

[marjotim]

Sometimes, seemingly randomly, running a build in Bamboo fails.
Build jobs/stages are run inside docker, failure seems to be coming randomly, regardless of agent (most of the time it succeeds in all agents, sometimes fails in any/some agents that have built it successfully before and do build successfully after).

Log snippet from failing build:

simple 13-Sep-2023 14:59:19 Creating local git repository in '/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git'.
simple 13-Sep-2023 14:59:19 /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git: Permission denied
error 13-Sep-2023 14:59:19 Cannot fetch branch '(unresolved) HashAndSource{refName=[null], branch=[bugfix/TEST-6503], hash=[36d68f0c4f89937ef79615144625d2c23181af06]}' from 'ssh://git@git.theserver.org:7999/example/project.git' to source directory '/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6'. command [/usr/bin/git init] failed with code 1. Working directory was [/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6]., stderr:/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git: Permission denied
simple 13-Sep-2023 14:59:19 Warning: failed to checkout source code to directory '/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6', trying to recover...
simple 13-Sep-2023 14:59:19 Cleaned source directory '/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6'...
simple 13-Sep-2023 14:59:19 Creating local git repository in '/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git'.
simple 13-Sep-2023 14:59:19 Initialized empty Git repository in /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git/
simple 13-Sep-2023 14:59:19 Fetching 'refs/heads/*' from 'ssh://git@git.theserver.org:7999/example/project.git'.
simple 13-Sep-2023 14:59:19 Warning: Permanently added '[127.0.0.1]:38017' (RSA) to the list of known hosts.
simple 13-Sep-2023 14:59:19 From ssh://127.0.0.1:38017/example/project
simple 13-Sep-2023 14:59:19 * [new branch] TEST-2100 -> TEST-2100
simple 13-Sep-2023 14:59:19 * [new branch] TEST-2285 -> TEST-2285
simple 13-Sep-2023 14:59:19 * [new branch] TEST-2648 -> TEST-2648
simple 13-Sep-2023 14:59:19 * [new branch] TEST-2652 -> TEST-2652
simple 13-Sep-2023 14:59:19 * [new branch] TEST-2681 -> TEST-2681
simple 13-Sep-2023 14:59:19 * [new branch] TEST-292 -> TEST-292
simple 13-Sep-2023 14:59:19 * [new branch] TEST-2969 -> TEST-2969
simple 13-Sep-2023 14:59:19 * [new branch] TEST-3541 -> TEST-3541
simple 13-Sep-2023 14:59:19 * [new branch] TEST-421 -> TEST-421
simple 13-Sep-2023 14:59:19 * [new branch] TEST-437 -> TEST-437
simple 13-Sep-2023 14:59:19 * [new branch] TEST-4937 -> TEST-4937
simple 13-Sep-2023 14:59:19 * [new branch] TEST-505 -> TEST-505
simple 13-Sep-2023 14:59:19 * [new branch] TEST-5686 -> TEST-5686
simple 13-Sep-2023 14:59:19 * [new branch] example-message -> example-message
simple 13-Sep-2023 14:59:19 * [new branch] example-testing -> example-testing
simple 13-Sep-2023 14:59:19 * [new branch] bugfix/TEST-2697 -> bugfix/TEST-2697
simple 13-Sep-2023 14:59:19 * [new branch] bugfix/TEST-4744 -> bugfix/TEST-4744
simple 13-Sep-2023 14:59:19 * [new branch] bugfix/TEST-6503 -> bugfix/TEST-6503
simple 13-Sep-2023 14:59:19 * [new branch] continuous_ver_numbering -> continuous_ver_numbering
simple 13-Sep-2023 14:59:19 * [new branch] master -> master
simple 13-Sep-2023 14:59:19 * [new tag] EX_1.0.3.0 -> EX_1.0.3.0
simple 13-Sep-2023 14:59:19 * [new tag] EX_2.0.0.0 -> EX_2.0.0.0
simple 13-Sep-2023 14:59:19 * [new tag] EX_2.1.1.0 -> EX_2.1.1.0
simple 13-Sep-2023 14:59:19 * [new tag] EX_2.2.1.0 -> EX_2.2.1.0
simple 13-Sep-2023 14:59:19 * [new tag] EX_2.5.1.0 -> EX_2.5.1.0
simple 13-Sep-2023 14:59:19 * [new tag] EX_2.6.0.0 -> EX_2.6.0.0
simple 13-Sep-2023 14:59:19 * [new tag] EX_2.6.1.0 -> EX_2.6.1.0
simple 13-Sep-2023 14:59:19 * [new tag] EX_2.7.1.0 -> EX_2.7.1.0
simple 13-Sep-2023 14:59:19 Fetching to source directory '/home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6' completed, proceeding with checkout...
simple 13-Sep-2023 14:59:19 Checking out revision 36d68f0c4f89937ef79615144625d2c23181af06.
simple 13-Sep-2023 14:59:19 Switched to branch 'bugfix/TEST-6503'
simple 13-Sep-2023 14:59:19 Checkout completed, recover successful.
simple 13-Sep-2023 14:59:19 Updated source code to revision: 36d68f0c4f89937ef79615144625d2c23181af06
simple 13-Sep-2023 14:59:19 Finished task 'Source Code Checkout' with result: Success
simple 13-Sep-2023 14:59:19 Preparing artifact 'DependencyCheckArtifacts' for use at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 14:59:19 Artifact 'DependencyCheckArtifacts' prepared in 56.82 ms at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6
simple 13-Sep-2023 14:59:19 Preparing artifact 'test_report.xml' for use at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 14:59:19 Artifact 'test_report.xml' prepared in 11.09 ms at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/test_report.xml
simple 13-Sep-2023 14:59:19 Preparing artifact 'testcoverage' for use at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 14:59:19 Artifact 'testcoverage' prepared in 11.09 ms at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/testcoverage.out
simple 13-Sep-2023 14:59:19 Preparing artifact 'testoutput.json' for use at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 14:59:19 Artifact 'testoutput.json' prepared in 13.88 ms at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/testoutput.json
simple 13-Sep-2023 14:59:19 Preparing artifact 'testoutput.out' for use at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 14:59:19 Artifact 'testoutput.out' prepared in 23.07 ms at /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/testoutput.out
simple 13-Sep-2023 14:59:19 Running pre-build action: Build Log Labeller Pre Build Action
simple 13-Sep-2023 14:59:19 Running pre-build action: VCS Version Collector
simple 13-Sep-2023 14:59:19 Starting task 'Script' of type 'com.atlassian.bamboo.plugins.scripttask:task.builder.script'
command 13-Sep-2023 14:59:19 Beginning to execute external process for build 'MW-Translation layer - example - bugfix-TEST-6503 - Build Service #10 (EXAMPLE-PROJECT21-JOB6-10)'\n ... running command line: \n/tmp/runInDocker5989393067675107037.sh /home/bamboo/bamboo-agent-home/temp/EXAMPLE-PROJECT21-JOB6-10-ScriptBuildTask-5932330718185466550.sh\n ... in: /tmp\n
[...]
build 13-Sep-2023 14:59:20 Creating version file
error 13-Sep-2023 14:59:20 fatal: Unable to read current working directory: No such file or directory
error 13-Sep-2023 14:59:20 mkdir: cannot create directory 'artifacts': No such file or directory
[...]
simple 13-Sep-2023 14:59:20 Publishing an artifact: version
error 13-Sep-2023 14:59:20 Unable to publish artifact [version]: the source directory /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/artifacts does not exist.
simple 13-Sep-2023 14:59:20 The artifact hasn't been successfully published after 1.210 ms

The first error ("fatal: unable to read current working directory") comes from running git tag.
As said, this seems to occur randomly, regardless of the agent that runs this job. It's as if current working directory gets removed (or removed and re-created - I'm in Ubuntu).

Most of the time it goes without problems, log snippet from a successful build:

simple 13-Sep-2023 16:02:57 Checking out into /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6
simple 13-Sep-2023 16:02:57 Updating source code to revision: 36d68f0c4f89937ef79615144625d2c23181af06
simple 13-Sep-2023 16:02:57 Creating local git repository in '/home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git'.
simple 13-Sep-2023 16:02:57 Initialized empty Git repository in /var/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git/
simple 13-Sep-2023 16:02:57 Fetching 'refs/heads/*' from 'ssh://git@git.theserver.org:7999/example/project.git'.
simple 13-Sep-2023 16:02:57 Warning: Permanently added '[127.0.0.1]:45915' (RSA) to the list of known hosts.
simple 13-Sep-2023 16:02:57 From ssh://127.0.0.1:45915/example/project
simple 13-Sep-2023 16:02:57 * [new branch] TEST-2100 -> TEST-2100
simple 13-Sep-2023 16:02:57 * [new branch] TEST-2285 -> TEST-2285
simple 13-Sep-2023 16:02:57 * [new branch] TEST-2648 -> TEST-2648
simple 13-Sep-2023 16:02:57 * [new branch] TEST-2652 -> TEST-2652
simple 13-Sep-2023 16:02:57 * [new branch] TEST-2681 -> TEST-2681
simple 13-Sep-2023 16:02:57 * [new branch] TEST-292 -> TEST-292
simple 13-Sep-2023 16:02:57 * [new branch] TEST-2969 -> TEST-2969
simple 13-Sep-2023 16:02:57 * [new branch] TEST-3541 -> TEST-3541
simple 13-Sep-2023 16:02:57 * [new branch] TEST-421 -> TEST-421
simple 13-Sep-2023 16:02:57 * [new branch] TEST-437 -> TEST-437
simple 13-Sep-2023 16:02:57 * [new branch] TEST-4937 -> TEST-4937
simple 13-Sep-2023 16:02:57 * [new branch] TEST-505 -> TEST-505
simple 13-Sep-2023 16:02:57 * [new branch] TEST-5686 -> TEST-5686
simple 13-Sep-2023 16:02:57 * [new branch] example-message -> example-message
simple 13-Sep-2023 16:02:57 * [new branch] example-testing -> example-testing
simple 13-Sep-2023 16:02:57 * [new branch] bugfix/TEST-2697 -> bugfix/TEST-2697
simple 13-Sep-2023 16:02:57 * [new branch] bugfix/TEST-4744 -> bugfix/TEST-4744
simple 13-Sep-2023 16:02:57 * [new branch] bugfix/TEST-6503 -> bugfix/TEST-6503
simple 13-Sep-2023 16:02:57 * [new branch] continuous_ver_numbering -> continuous_ver_numbering
simple 13-Sep-2023 16:02:57 * [new branch] master -> master
simple 13-Sep-2023 16:02:57 * [new tag] EX_1.0.3.0 -> EX_1.0.3.0
simple 13-Sep-2023 16:02:57 * [new tag] EX_2.0.0.0 -> EX_2.0.0.0
simple 13-Sep-2023 16:02:57 * [new tag] EX_2.1.1.0 -> EX_2.1.1.0
simple 13-Sep-2023 16:02:57 * [new tag] EX_2.2.1.0 -> EX_2.2.1.0
simple 13-Sep-2023 16:02:57 * [new tag] EX_2.5.1.0 -> EX_2.5.1.0
simple 13-Sep-2023 16:02:57 * [new tag] EX_2.6.0.0 -> EX_2.6.0.0
simple 13-Sep-2023 16:02:57 * [new tag] EX_2.6.1.0 -> EX_2.6.1.0
simple 13-Sep-2023 16:02:57 * [new tag] EX_2.7.1.0 -> EX_2.7.1.0
simple 13-Sep-2023 16:02:57 Checking out revision 36d68f0c4f89937ef79615144625d2c23181af06.
simple 13-Sep-2023 16:02:57 Switched to branch 'bugfix/TEST-6503'
simple 13-Sep-2023 16:02:57 Updated source code to revision: 36d68f0c4f89937ef79615144625d2c23181af06
simple 13-Sep-2023 16:02:57 Finished task 'Source Code Checkout' with result: Success
simple 13-Sep-2023 16:02:57 Preparing artifact 'DependencyCheckArtifacts' for use at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 16:02:57 Artifact 'DependencyCheckArtifacts' prepared in 56.76 ms at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6
simple 13-Sep-2023 16:02:57 Preparing artifact 'test_report.xml' for use at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 16:02:58 Artifact 'test_report.xml' prepared in 11.50 ms at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/test_report.xml
simple 13-Sep-2023 16:02:58 Preparing artifact 'testcoverage' for use at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 16:02:58 Artifact 'testcoverage' prepared in 16.33 ms at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/testcoverage.out
simple 13-Sep-2023 16:02:58 Preparing artifact 'testoutput.json' for use at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 16:02:58 Artifact 'testoutput.json' prepared in 9.364 ms at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/testoutput.json
simple 13-Sep-2023 16:02:58 Preparing artifact 'testoutput.out' for use at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6 (location: )
simple 13-Sep-2023 16:02:58 Artifact 'testoutput.out' prepared in 8.950 ms at /home/bamboo-agent/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/testoutput.out
simple 13-Sep-2023 16:02:58 Running pre-build action: Build Log Labeller Pre Build Action
simple 13-Sep-2023 16:02:58 Running pre-build action: VCS Version Collector
simple 13-Sep-2023 16:02:58 Starting task 'Script' of type 'com.atlassian.bamboo.plugins.scripttask:task.builder.script'
command 13-Sep-2023 16:02:58 Beginning to execute external process for build 'MW-Translation layer - example - bugfix-TEST-6503 - Build Service #11 (EXAMPLE-PROJECT21-JOB6-11)'\n ... running command line: \n/tmp/runInDocker858592673198618565.sh /home/bamboo-agent/bamboo-agent-home/temp/EXAMPLE-PROJECT21-JOB6-11-ScriptBuildTask-5747922812257384322.sh\n ... in: /tmp\n
[...]
simple 13-Sep-2023 16:03:38 Publishing an artifact: version
simple 13-Sep-2023 16:03:38 Finished publishing of artifact Required shared artifact: [version], pattern: [version*] anchored at: [artifacts] in 12.65 ms

Any ideas what causes this and how to fix it?
We're running Atlassian Bamboo version 8.1.3 build 80110

[Khushboo Gupta · Accepted solution]

Hello @marjotim 

The issues you're encountering with Bamboo builds failing intermittently due to permission errors and "unable to read current working directory" problems can stem from several potential causes. Here are some steps to diagnose and possibly resolve these issues.

1. Permissions and Ownership Issues

The errors like /home/bamboo/bamboo-agent-home/xml-data/build-dir/EXAMPLE-PROJECT21-JOB6/.git: Permission denied suggest there might be issues with permissions or ownership of the directories where Bamboo tries to perform operations. This can happen if the Bamboo agent processes are not consistently running as the same user or if external scripts change ownership of the files.

Fix:

• Ensure that the user running the Bamboo agent has consistent and appropriate permissions to access and modify the directories and files within /home/bamboo/bamboo-agent-home/xml-data/build-dir/.

Since your builds are running inside Docker containers, it's possible that inconsistencies or issues with Docker volumes or mounts are causing these problems. Docker might not be mounting volumes consistently, or the Docker image might not have the right environment setup.

• Check the Docker configurations for volume mounts to ensure they are set up consistently across different agents and builds.
• Consider using Docker's --user option to set the UID and GID that matches the host, to avoid permissions issues with mounted volumes.

Race Conditions and Directory Structure Changes

Given that it occurs randomly and seemingly regardless of the agent that runs the job, it could be a race condition, where multiple processes are trying to access or modify the working directory at the same time. This can sometimes lead to unpredictable results, like the directory seeming to disappear temporarily.

It could be Antivirus which may lock some of the files while scanning them and prevent Bamboo from updating/refreshing the cache. Hence Disable the continuous monitoring and scans for the Bamboo directories. Refer : https://confluence.atlassian.com/bamkb/frequent-git-operations-failing-due-to-corrupted-git-cache-on-windows-1101930410.html 

Let me know if this helps!

Regards,

Khushboo Gupta

*please don't forget to Accept the answer if your query was answered**

[marjotim]

Thanks, the issue was actually resolved through a support request last October - I had completely forgot I had also posted a question. (Root cause was directory permission that had gone wrong when cloning Bamboo agent).