Multiple self-hosted runners on a MacOS

[Andrew Welty]

I've been struggling this week reconfiguring my existing, working bitbucket cloud pipeline to run multiple simultaneous builds.  This pipeline has 3 steps, but only the first step, the build, is important for this issue (I think).  It's building a .Net 8.0 IOS C# Xamarin app on a local MacOS, using a self-hosted runner.  I need the ability to run this pipeline multiple times simultaneously.  I can kick off individual pipelines with the Bitbucket API, that's not an issue at all.  With only one runner, running on that self-hosted MacOS, it works perfectly.  When I add another runner, and run it on the same MacOS, things go badly.  I've got 3 configured, currently, each running in its own folder.  I'm setting each one's workingDirectory in the start.sh command, and there shouldn't be anything in the yaml file that contradicts or interferes with that folder.  And yet, when they run, they ALWAYS fail, with a System Error that complains about a folder that is definitely outside the folder set in the workingDirectory.  Analysis of the runner logs mention that the runner couldn't find the sms folder or artifact folder, and there, it lists the correct directory (but again, can't find subfolders, and I believe it should be creating those subfolders, but is failing.)  I've verified the user has full access rights to the folder structure in question.
Upon request, I can provide logs and the yaml file, though I will need to scrub them of company-sensitive information.  At this stage, I'll take any insight I can get.

[Patrik S · Accepted solution]

Hello @Andrew Welty ,

and welcome to the Community!

When using MacOS self-hosted runners with Bitbucket Pipelines, it's essential to remember that these runners execute directly within the machine's shell environment. Unlike Docker runners, they don't provide isolation, so all builds share the same system environment.

Because of this shared environment, deploying multiple runners on a single MacOS machine can lead to conflicts and resource contention. If multiple runners are running simultaneously, they might interfere with each other, especially if they modify or access shared resources on the system. For example, if a build script installs a new library or changes a system configuration, that modification will affect all runners and subsequent builds on the machine.

To mitigate this, I would suggest ensuring that your build process does not have any shared resources or state between the different builds. For instance, if your builds write to global directories or use global configurations, they might interfere with each other. 

Also, verify that your build tools (like Xamarin, .NET SDK) are not configured to use global settings or caches that could cause conflicts.

If feasible, you can also consider further isolating builds using virtual environments. This can prevent any cross-contamination between different build jobs.

I hope this helps to shed some light on the investigation and what could be causing it. Should you have any questions, please let us know.

Thank you, @Andrew Welty .

Patrik S

[Andrew Welty]

Thanks for your answer, Patrick!  This is helpful, although I did know most of this.  I'm not really getting far enough to troubleshoot most of that, however, because the pipeline isn't even getting started:

The text underneath reads "/Users/[username]/Desktop/MacOsRunner2/atlassian-bitbucket-pipelines-runner/bin/../temp/[runnerguid]/build".  Importantly, that should NOT be the folder that runner is running in.  It SHOULD be "/Users/[username]/Desktop/GitProjects/[projectname]/MacOsRunner/Runner1/atlassian-bitbucket-pipelines-runner/bin/../temp/[runnerguid]/build"

Sometimes, when it runs, that "MacOsRunner2" is a different number - generally 1 or no number there at all, simply "MacOsRunner".

Interestingly, the runner guid is correct - that's how I'm matching up the pipeline attempts with the actual correct folders on the mac.  And at the same time, I do get runner logs on the mac, in the correct folder.  Those logs consistently contain a "NoSuchFileException" like this one:
[2024-11-26 11:57:21,180] Setting up directories.
[2024-11-26 11:57:21,188] An error occurred whilst setting up directories.
java.nio.file.NoSuchFileException: /Users/[username]/Desktop/MacOsRunner/atlassian-bitbucket-pipelines-runner/bin/../temp/[runnerguid]/ssh
Some of them complain about an "artifact" folder, in place of that ssh folder.  The fact that this is happening while setting up directories implies to me that the runner should be creating these folders.  Even so, I've attempted manually creating them, and granting full rights to that path (despite the fact that the path is outside where I think it should be running to begin with), to no avail.  The error persists.  It truly feels like the runner is just doing something that I don't understand.  And remember, to me, that runner is a black box.  I can't see inside it or get details on what it's doing.

Remember, when it wrote that log, the runner itself was running in "/Users/[username]/Desktop/GitProjects/[projectname]/MacOsRunner/Runner1/atlassian-bitbucket-pipelines-runner/bin/../temp/[runnerguid]/build", and should be configured to use that folder as its "home".

One of those "shared environments" must be dictating the folder location somehow.  I can't identify anything anywhere in my configuration or pipeline where I'm dictating that path, so it must be something the runner is coming up with.  If I could get further, I could troubleshoot the shared environment situation.  (Honestly, right now I don't think this pipeline writes to ANY shared folders, the intention is to keep each runner separate so that doesn't happen.)

I would appreciate any help figuring out how and why the runner is "choosing" the wrong path here.

[Patrik S]

Hello @Andrew Welty ,

Thanks for providing additional details.

It's puzzling that the folder you're seeing is not the correct one you've set as the working directory. I'd like to ask for additional questions to better understand your setup:

• Have you ever created a runner in that folder in the past (/Users/[username]/Desktop/MacOsRunner2/atlassian-bitbucket-pipelines-runner/bin/../temp/[runnerguid])?
• Could it be the case that you still have runners you might have created in the past still running on the machine? I would recommend killing all java processes repeated to runners and then start the runner from scratch to do the test.
• What is the exact command you are using to start the runner (when sharing the command, please remove any sensitive data such as the OAuth client ID and secret, but include the working directory you have set). Please share the command you have used to start both runners.
• Have you created a single runner and re-used the same command to start multiple runners, or are you creating a new runner in the UI and using the individual given command to start a new runner instance? 
• Would it be possible for you to test in a different machine and check if that's reproducible ?

[Andrew Welty]

Great minds think alike, I was already composing this:

This is my macosrunner.sh file, for one of my runners, with sensitive info redacted:
#!/bin/bash
export BITBUCKET_RUNNER_HOME="/Users/[username]/Desktop/GitProjects/[projectname]/MacOsRunner/Runner1"
cd "/Users/[username]/Desktop/GitProjects/[projectname]/MacOsRunner/Runner1/atlassian-bitbucket-pipelines-runner/bin"
./start.sh --accountUuid {[guid]} --repositoryUuid {[guid]} --runnerUuid {[guid]} --OAuthClientId [redacted] --OAuthClientSecret [redacted] --runtime macos-bash --workingDirectory /Users/[username]/Desktop/GitProjects/[projectname]/MacOsRunner/Runner1/temp

I'm setting BITBUCKET_RUNNER_HOME and workingDirectory explicitly to the correct directory.
My YAML file shouldn't be doing anything to change those values.

I currently have 3 of these, each with its own runner guid and "RunnerX" folder.  Otherwise, all 3 macosrunner.sh files are identical.

To answer your other questions:
Yes, I believe I DID previously have other runners running in that "incorrect" folder.  I will attempt to kill any java processes, to clean that up and try again.
I believe I've tried just one of these at a time, and it does work fine when running alone.  I'll try that test again, to be sure.
And my research indicated that each simultaneously running pipeline needs its own runner.  The number of runners I have will also be the number of pipelines I can run at once.  As such, I'm configuring multiple runners in the UI, and using each runner's specific information (the runner guid is the only thing that changes between them) when running them on the mac.

Oh, and sadly, we only have one mac to use for testing.

[Andrew Welty]

I'm pleased to report that killing all java processes on the mac has worked.  Apparently, something from the prior runner(s) was holding onto a path in the environment, despite having already killed the runner terminals.  We will remember this for the future!
I'm still facing issues in the pipeline, but I believe those are related to the shared resources we've discussed.
I'm accepting this answer at this time, because it seems to have resolved the issue, but if you care to answer, I do have another question, on the same topic.

I understand the runners share an environment.  I understand that to mean a larger environment in which the runners are running - .net SDKs, the compiler, and any environment variables are all good examples.  However, I'm curious about the repo itself.  The repo is being cloned during pipeline initialization, so I assume I'm getting 3 copies of the repo on this mac, but I'd like explicit confirmation of this.
Thanks!

[Patrik S]

Hey @Andrew Welty ,

Good to hear that killing the Java processes did the trick :)

As for your last question, you're correct: each runner will clone its own copy of the repository when running the build, and place that copy in that runner's temp folder (the folder with the runner's UUID) within the working directory.