Cleanup Issues with Symlinks in Self-Hosted Runners' Working Directory

[Stefan Hirschenberger]

Hello Community,

We are currently facing a challenge with our self-hosted Windows runner used in our CI/CD pipelines. Our setup involves using Gitman to manage dependencies, which includes creating symlinks within the working directory. Specifically, these symlinks point to other folders within the same working directory. It appears that the cleanup issues occur when the target folder of a symlink is deleted before the symlink itself (?), leading to NoSuchFileException errors- subsequently to an unhealthy state of the runner.

Here’s a snippet of the log showing the issue:

[2024-07-29 22:15:13,432] Appending log line to main log.
[2024-07-29 22:15:13,763] Tearing down directories.
[2024-07-29 22:15:14,083] An error occurred whilst tearing down directories.
java.nio.file.NoSuchFileException: C:\<path_to_symlink_in_working_dir>\BurDefaultLibraries...

Currently, we manually delete the symlinks in the after-script section of our pipeline, which is a workaround but not an ideal solution. It adds complexity and reduces the readability of our pipeline code.

Has anyone else encountered similar issues with symlinks in self-hosted runners? Any insights or recommendations on how to handle or prevent these errors more gracefully would be greatly appreciated.

Thank you in advance for your help and suggestions!

[Theodora Boudale · Accepted solution]

Hi Stefan,

Thank you for reaching out. I was able to reproduce this on a repo of mine with the latest version of the Windows runner. Please allow me some time to discuss it with my team, and I'll get back to you on this.

Kind regards,
Theodora

[Theodora Boudale]

Hi Stefan,

I discussed this with my team and it looks like a bug with the runner. I was able to reproduce this, but only when the name of the symlink comes after the name of the directory the symlink is pointing to in alphabetical order.

I created a bug report for our development team to look into this:

• https://jira.atlassian.com/browse/BCLOUD-23347

You can add yourself as a watcher (by selecting the Start watching this issue link in the bug report) if you'd like to get notified via email on any updates. You can also select the This affects my team link to indicate you are affected by this issue. Please note though that we don't provide ETAs and bug fixing is implemented as per our policy here.

In the meantime, deleting the symlinks in the after-script is the only workaround I found as well.

Kind regards,
Theodora

[Stefan Hirschenberger]

If someone stumbles across this one:

Here’s a Bash script to remove the symlinks in the after script section:

find . -type l -exec rm -v {} \;

Limitation:
The approach fails if the pipeline is stopped manually, as the after-script is not executed in this case.
Therefore, the runner will transition to an unhealthy state.