Self-hosted runner: 409 Error when updating runner state to ONLINE

[Crylion]

I am running a workspace runner instance on a Mac Studio in our office, and for some reason the runner is just randomly unable to update its status because of an error.

HttpResponseSummary{httpStatusCode=409, httpStatusMessage=Conflict, bodyAsString={"key":"agent-service.runner.conflict","message":"Simultaneous state updates were attempted for runner with id: {57ab238e-32cf-5847-bc02-87d122791d4a}","arguments":{}}}

This is pretty puzzling to me, because it generally workds and successfully updates its state every 30 seconds for hours and then randomly runs into this connection error a few times. So the "ONLINE" state of our runner just sort of fluctuates and the problem seems to get worse, the longer the runner is online since the last restart of the service.

[2023-03-27 15:29:44,104] Updating runner status to "ONLINE" and checking for new steps assigned to the runner after 0 seconds and then every 30 seconds.

[2023-03-27 15:29:44,108] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,813] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,818] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,825] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,829] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,833] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,836] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,837] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,838] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,840] Updating runner state to "ONLINE".

[2023-03-27 15:35:05,840] [e97e1f32-2, L:/192.168.2.202:52923 - R:api.atlassian.com/185.166.143.32:443] The connection observed an error, the request cannot be retried as the headers/body were sent

java.io.IOException: Connection reset by peer

 at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)

 at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

 at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)

 at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)

 at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)

 at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)

 at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:258)

 at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)

 at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)

 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)

 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)

 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)

 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)

 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)

 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

 at java.base/java.lang.Thread.run(Thread.java:829)

[2023-03-27 15:35:05,841] Updating runner state to "ONLINE".

[2023-03-27 15:35:06,102] An error occurred whilst updating runner state to "ONLINE".

com.atlassian.pipelines.stargate.client.core.exceptions.StargateConflictException: Response Summary: HttpResponseSummary{httpStatusCode=409, httpStatusMessage=Conflict, bodyAsString={"key":"agent-service.runner.conflict","message":"Simultaneous state updates were attempted for runner with id: {57ab238e-32cf-5847-bc02-87d122791d4a}","arguments":{}}}

 at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

 Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 

Error has been observed at the following site(s):

 *__checkpoint ⇢ 409 from PUT https://api.atlassian.com/ex/bitbucket-pipelines/rest/internal/accounts/%7B6e3b2af8-8ed2-4e9d-ac3a-1af0bd77694d%7D/runners/%7B57ab238e-32cf-5847-bc02-87d122791d4a%7D/state [DefaultWebClient]

Original Stack Trace:

 at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

 at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

 at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

 at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)

 at com.atlassian.bitbucketci.client.reactive.ResponseExceptionFactory$ConstructorInvoker.invokeConstructor(ResponseExceptionFactory.java:125)

 at io.vavr.CheckedFunction1.lambda$unchecked$43b513dd$1(CheckedFunction1.java:220)

 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)

 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)

 at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)

 at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)

 at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)

 at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)

 at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)

 at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)

 at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)

 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)

 at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)

 at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)

 at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473)

 at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:702)

 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)

 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)

 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)

 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)

 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)

 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)

 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519)

 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458)

 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)

 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)

 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)

 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)

 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)

 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

 at java.base/java.lang.Thread.run(Thread.java:829)

Since we only have this one runner on this machine and pretty much nothing else, it seems confusing, why this error should occur.

[Bob Brand]

I am also seeing the same error. 

I'm on a local windows runner.

The runner version is the latest, 1.518

2023-10-30 21:23:47,071] An error occurred whilst updating runner state to "ONLINE".
com.atlassian.pipelines.stargate.client.core.exceptions.StargateConflictException: Response Summary: HttpResponseSummary{httpStatusCode=409, httpStatusMessage=Conflict, bodyAsString={"key":"agent-service.runner.conflict","message":"Simultaneous state updates were attempted for runner with id: {d5a927f0-98c7-5633-b67a-6e62d2b1fa29}","arguments":{}}}
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint Γçó 409 from PUT https://api.atlassian.com/ex/bitbucket-pipelines/rest/internal/accounts/%7Bc3fa9b68-7539-4dac-805c-94417d08a52a%7D/runners/%7Bd5a927f0-98c7-5633-b67a-6e62d2b1fa29%7D/state [DefaultWebClient]

[Bob Brand]

FWIW, I reverted the runner software to 1.508 and I am not seeing the problem after.

[Oday Rafeh]

Hi @Crylion 

The 409 error you are experiencing is caused by a conflict with a simultaneous state update for your runner. This can happen if multiple requests are trying to update the state of the same runner at the same time.

To troubleshoot this issue, you can try:

Verify that there is no other process or service running that may be conflicting with the state updates of the runner.

Restart the runner service and try updating the state again.

Check the network connection and ensure that it is stable and reliable.

Additionally, you may want to consider upgrading your runner to the latest version or using a different runner if the issue persists.