Self hosted Runners - An error occurred whilst updating runner state to "ONLINE"

[Erick Dzadotz]

Hi,

I have an autoscale runner configured, but when the runner finishes running it does not descale or go ONLINE to run other pipelines or steps.

The Runner executes, it stays ONLINE in the bitbucket workspace, the pods scale to exec the parallel steps and after a while the error occurs and the pods are not descaled.

My configs:

constants:
  default_sleep_time_runner_setup: 5
  default_sleep_time_runner_delete: 5
  runner_api_polling_interval: 60
  runner_cool_down_period: 10

strategy: "percentageRunnersIdle"
min: 1 
max: 10 
scale_up_threshold: 0.9 
scale_down_threshold: 0.8 
scale_up_multiplier: 1.5 
scale_down_multiplier: 0.3 

resources: 
  requests:
    memory: "1Gi"
    cpu: "100m"
  limits:
    memory: "1Gi"
    cpu: "1000m"

The error of pods that not works:

[2023-11-09 16:26:16,939] An error occurred whilst updating runner state to "ONLINE".
com.atlassian.pipelines.stargate.client.core.exceptions.StargateUnauthorizedException: Response Summary: HttpResponseSummary{httpStatusCode=401, httpStatusMessage=Unauthorized, bodyAsString={"code":401,"message":"Unauthorized"}}
at jdk.internal.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ 401 from PUT https://api.atlassian.com/ex/bitbucket-pipelines/rest/internal/accounts/123456foo123456/runners/%7B02887258-36ab-5c50-9bea-XPTO/state [DefaultWebClient]
Original Stack Trace:
at jdk.internal.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.atlassian.bitbucketci.client.reactive.ResponseExceptionFactory$ConstructorInvoker.invokeConstructor(ResponseExceptionFactory.java:125)
at io.vavr.CheckedFunction1.lambda$unchecked$43b513dd$1(CheckedFunction1.java:220)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:424)
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:478)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:712)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)

[Theodora Boudale]

Hi Erick and welcome to the community!

Are you using a Runner Autoscaler for Kubernetes as per this guide?

• https://community.atlassian.com/t5/Bitbucket-Pipelines-Runner/Bitbucket-Pipelines-Runner-Autoscaler-for-Kubernetes-Getting/ba-p/1989654

If so, I would suggest joining the following community group and asking your question there, as this group is specifically for Runner Autoscaler questions:

• https://community.atlassian.com/t5/Bitbucket-Pipelines-Runner/gh-p/bitbucket-runner-autoscaler-4k8s

Kind regards,
Theodora

[Erick Dzadotz]

Hi Theodora,

Thank you, I sent a request to join there. I'm using Runner Autoscaler for Kubernetes following this procedure:

• https://bitbucket.org/bitbucketpipelines/runners-autoscaler/src/master/

After I receive approval to join the group I can check your first link. For now I don't have sufficient privileges.

Thanks,

Erick