macOS runner 3.6 crashes all the time

[MennoVink]

I'm trying to get a self-hosted macos runner up. Every pipeline i start the runner crashes. The runner experiences a java crash during the clone step.

[2024-11-20 16:06:26,531] Updating step progress to CLONING. [2024-11-20 16:06:26,761] An error occurred whilst updating step progress. com.atlassian.pipelines.stargate.client.core.exceptions.StargateForbiddenException: Response Summary: HttpResponseSummary{httpStatusCode=403, httpStatusMessage=Forbidden, bodyAsString={"error":{"message":"Forbidden","detail":"Runner currently has no pipeline scheduled.","data":{"key":"rest-service.rest-service.forbidden","arguments":{}}}}} at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): *__checkpoint ⇢ 403 from POST https://api.atlassian.com/ex/bitbucket-pipelines/rest/1.0/accounts/REMOVED/progressUpdate [DefaultWebClient] Original Stack Trace: at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:501) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:485) at com.atlassian.bitbucketci.client.reactive.ResponseExceptionFactory$ConstructorInvoker.invokeConstructor(ResponseExceptionFactory.java:125) at io.vavr.CheckedFunction1.lambda$unchecked$43b513dd$1(CheckedFunction1.java:220) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839) at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160) at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103) at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103) at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415) at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439) at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493) at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:789) at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1349) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1389) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1575)

[Theodora Boudale · Accepted solution]

Hi,

We have released a newer version of the runner, 3.7.0. I suggest installing this newer version first, as version 3.6.0 had a very short timeout configured which can cause issues with artifacts. You don't need to create a new runner from the UI, you can simply remove the existing runner folder from your Mac (after stopping the runner), then download the newer version and install it at the same location:

curl https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner-3.7.0.tar.gz --output atlassian-bitbucket-pipelines-runner.tar.gz

mkdir atlassian-bitbucket-pipelines-runner && tar -xzvf atlassian-bitbucket-pipelines-runner.tar.gz -C atlassian-bitbucket-pipelines-runner

cd atlassian-bitbucket-pipelines-runner/bin

Then, use the preconfigured command we provided when you created the runner in order to start it.

If you are still experiencing issues, can you please let me know the following:

1. Do you have additional runners, either on this Mac or on a different Mac? Or did you start multiple instances of the same runner either on this or on a different Mac?

2. When a pipeline starts, do you see a build listed in the Pipelines page of the repo on our website?

If so, please open that build, and expand the section Build setup in the log. Are there any errors listed there? Can you please copy-paste here the content of the section? (Please make sure you sanitize private data before sharing, if any).

3. Does the Mac meet the following minimum requirements for MacOS runners?

• MacOS Catalina 10.15+
• OpenJDK 11 (11.0.11 or newer) is installed
• Git 2.35.0 and above
• A 64-Bit MacOS instance with at least 8GB of RAM as a host for the runner.

Kind regards,
Theodora

[MennoVink]

@Theodora Boudale Runner 3.7 also crashes.

1. I do have another runner, one running on windows. I'm migrating mac intel to mac arm. I used to have a runner on the intel mac but that's no longer active. I have not managed to get any runner up on this arm mac. The windows pipeline happily clones, builds and uploads artefacts.
2. Runner 3.7 crashes during startup on macos, as such it is showing as OFFLINE in the runners page. A pipeline does get started, the stage requiring the macos label is reported as 'Currently no online runner available that matches the required labels.'
3. The mac is a new Apple M4.
   
   macos 15.1.1
   
   java --version
   openjdk 23.0.1 2024-10-15
   OpenJDK Runtime Environment Homebrew (build 23.0.1)
   OpenJDK 64-Bit Server VM Homebrew (build 23.0.1, mixed mode, sharing)
   
   git --version
   git version 2.39.5 (Apple Git-154)

This is the crash log for runner 3.7:

[2024-11-27 18:13:30,275] Runner version: 3.7.0
[2024-11-27 18:13:30,279] Runner runtime: macos-bash
[2024-11-27 18:13:30,447] Starting websocket listening to RUNNER_UPDATED events.
[2024-11-27 18:13:30,491] Updating runner status to "ONLINE" and checking for new steps assigned to the runner after 0 seconds and then every 30 seconds.
[2024-11-27 18:13:30,547] Updating runner state to "ONLINE".
[2024-11-27 18:13:31,070] An error occurred whilst updating runner state to "ONLINE".
org.springframework.web.reactive.function.client.WebClientResponseException$BadRequest: 400 Bad Request from POST https://auth.atlassian.com/oauth/token
at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:217)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ 400 from POST https://auth.atlassian.com/oauth/token [DefaultWebClient]
Original Stack Trace:
at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:217)
at org.springframework.web.reactive.function.client.DefaultClientResponse.lambda$createException$1(DefaultClientResponse.java:207)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:103)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439)
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:789)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1349)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1389)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1575)

[Theodora Boudale]

Hi,

Thank you for the info.

We have released version 3.8.0 of the MacOS runner. Could you please update to 3.8.0 and let me know

1) Do you still get errors when starting the runner with the preconfigured command from the terminal, and if so, what errors?

2) If the runner does start and goes online, what happens when a build is triggered? If there are still errors, it would be useful to have the errors from the runner log and from the build's page on our website

The runner log is located at

<Path>/atlassian-bitbucket-pipelines-runner/temp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

where <Path> is the location where you unzipped the runner zip, and b609961f-891e-c872-c36b-f3f2c315d186 should be the UUID of your own runner.

You can get the newer version of the runner by using the same commands I shared before, only change the part 3.7.0 in the curl command to 3.8.0.

I also recommend restarting your Mac before updating to 3.8.0 to ensure that any processes from the previous runner versions are terminated.

Kind regards,
Theodora

[MennoVink]

I have now got the runner working again :)

For good measure to guarantee nothing was running anymore: i deleted all old runners, terminated all terminals and restarted the machine. I then created and installed the new 3.8 runner. And it comes online correctly.

So nice to have pipelines get picked up again. Thank you a lot for your assistance.

[Theodora Boudale]

It's good to hear that the runner is working now! Thank you for the update and you are very welcome.

Please feel free to reach out if you ever need anything else!