Self-hosted runner failing to get oauth token

[michael.wei]

I'm trying to setup a windows self-hosted runner so I can test the pipeline feature without using up all my available cloud minutes. There have been a few hurdles, the last one involving adding to the certificate store before I could get the runner online (corporate security things).

So I have a simple yml file that looks like this (using a global image):

pipelines:
  branches:
    master:
      - step:
          name: test step manual custom
          runs-on:
            - 'self.hosted'
            - 'windows'
          script:
            - echo "this should show it's working"

(This works fine if I remove the runs-on section.)

Upon execution, the output I get in my terminal looks like this (removing some secret key values)

[2022-04-01 17:29:10,781] Setting runner state to executing step.
[2022-04-01 17:29:10,834] Getting step StepId{accountUuid={REDACTED}, repositoryUuid={REDACTED}, pipelineUuid={REDACTED}, stepUuid={REDACTED}}.
[2022-04-01 17:29:10,875] Getting oauth token for step.
[2022-04-01 17:29:10,896] Getting environment variables for step.
[2022-04-01 17:29:11,811] An error occurred whilst getting oauth token.
com.atlassian.pipelines.stargate.client.core.exceptions.StargateBadRequestException: Response Summary: HttpResponseSummary{httpStatusCode=400, httpStatusMessage=Bad Request, bodyAsString=400 Bad Request}
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ 400 from POST https://api.atlassian.com/ex/bitbucket-pipelines/rest/1.0/accounts/%7B(accountUuid)%7D/repositories/%7B(repositoryUuid)%7D/oauth [DefaultWebClient]
Stack trace:
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.atlassian.bitbucketci.client.reactive.ResponseExceptionFactory$ConstructorInvoker.invokeConstructor(ResponseExceptionFactory.java:117)
at io.vavr.CheckedFunction1.lambda$unchecked$43b513dd$1(CheckedFunction1.java:220)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:684)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-04-01 17:29:11,847] Completing step with result Result{status=ERROR, error=Some(Error{key='runner.step.execution-error', message='Response Summary: HttpResponseSummary{httpStatusCode=400, httpStatusMessage=Bad Request, bodyAsString=400 Bad Request}', arguments={}})}.
[2022-04-01 17:29:12,265] Setting runner state to not executing step.
[2022-04-01 17:29:12,266] Waiting for next step.

All I can get from this is that the oauth token step is failing somehow, but no idea how to fix this.

[Xinyu Wang]

I have a similar situation. Any solution/update on this question?

[Patrik S]

Hello @Xinyu Wang ,

I would like to ask if you could please create a new question for your issue, providing details on the errors you see and also whether it concerns a Bitbucket Cloud or Bitbucket Server repo.

We generally encourage users to create a new question for their issue instead of posting on someone else’s question, because

1) the root cause and resolution may be different for each case

2) a question can become cluttered and difficult to follow if we try to troubleshoot multiple users’ issues in it.

Please feel free to let me know if you have any questions.

Thank you, @Xinyu Wang !

Patrik S

[Theodora Boudale]

Hi @michael_wei,

Based on the log, the issue seems to be with the request that the runner is making to our API.

What version of the runner are you using? If there is a build log for the build you tried to run in the Pipelines page of the repo, the Runner version should be visible in the Build log in the section Runner.

The latest version for the Windows Runner is 1.315

If you are running an older version, can you try downloading the latest one and let us know if the issue still occurs with that?

You can run the following commands to download the latest version, extract the zip and navigate to .\atlassian-bitbucket-pipelines-runner\bin

Invoke-WebRequest -Uri https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner-1.315.zip -OutFile .\atlassian-bitbucket-pipelines-runner.zip

Expand-Archive .\atlassian-bitbucket-pipelines-runner.zip

cd .\atlassian-bitbucket-pipelines-runner\bin

Then use the command you were given from Bitbucket UI to launch the runner.

Kind regards,
Theodora

[Michael Wei]

Interesting, I was apparently running version 1.313, but even after using the commands you gave to download 1.315 and extract, that still shows up as "Runner version: 1.313". I can't confirm this via build log as no build log is generated when this error occurs. The error still happens after using the commands you gave.

I will note that I tried setting up a runner on a different machine and it worked there. I suspect it must be due to how this machine's secured but I don't know how.

[Theodora Boudale]

Hi Michael,

I was apparently running version 1.313, but even after using the commands you gave to download 1.315 and extract, that still shows up as "Runner version: 1.313"

Were you perhaps launching the runner from the directory of the older version?

I would suggest stopping the runner, and then try deleting any existing

• atlassian-bitbucket-pipelines-runner.zip file
• atlassian-bitbucket-pipelines-runner directory

Then download again using the commands above and launch the runner.

Please feel free to let me know if the new version shows and also if running a build shows the same error in the log or not.

Kind regards,
Theodora

[Michael Wei]

Thanks Theo, sorry for the late reply; didn't get the notification.

My mistake; the script I used to hold the runner secrets was apparently changing to the old runner version's directory so I've fixed that and it now reports runner version 1.315 as expected.

But I'm still getting the error described in OP. Stack trace is virtually identical (just changes in line numbers, which makes sense with version change).

[Theodora Boudale]

Hi Michael,

Thanks for the update. I have asked our development team to take a look, I'll let you know as soon as I have an update.

Kind regards,
Theodora

[Theodora Boudale]

Hi Michael,

I am still waiting for feedback on this, I would just like to double-check a few things in the meantime:

1. Is this a workspace or a repository runner?
   
   
2. Can you confirm that the runner shows Online when you start it?
   
   Based on the output you posted here, it looks like it is online, but I just wanted to make sure.
   
   
3. What is the version of Windows on this server?
   
   
4. Are the following minimum requirements met for this server?
   
   - OpenJDK 11
   - Git 2.35.0 and above
   - PowerShell 5.0 and above
   - A 64-Bit Windows 10+ or Windows Server 2019+ instance with at least 8GB of RAM as a host for the runner.
   
   
5. I see in the output that the API URL that gives an error includes some parentheses (in bold below)

https://api.atlassian.com/ex/bitbucket-pipelines/rest/1.0/accounts/%7B(accountUuid)%7D/repositories/%7B(repositoryUuid)%7D/oauth

Do these parentheses actually show in the output of your terminal, or did you add them when redacting the values?

Kind regards,
Theodora

[Theodora Boudale]

Hi @michael_wei,

I spoke with the developers and we'll need to know the repo UUID as well as the accountUUID in order to investigate this. I went ahead and created a support ticket for you, so you can share this info there instead of here publicly.

You should have received an email with a link to the support ticket (I created the ticket using the email of the community account you used to create this question).

Just in case you haven't received the email, please let me know and I can post the link here (the ticket won't be visible to anyone other than you and Atlassian staff, even if they have the link).

Please share in the support ticket the repo UUID and the accountUUID (and also if the minimum requirements I mentioned are met for the server) and the dev team can further investigate.

If you have any questions, please feel free to let me know.

Kind regards,
Theodora

[Michael Wei]

Thanks Theo; appreciate the send to the original post account - I'm having difficulty accessing community with that account (it keeps logging me in as this other one).

I replied the information on that email you sent.

[Theodora Boudale]

Hi Michael,

You are very welcome. We received the info, replying to the email will also post a reply to the support ticket.

Regarding the issue with logging in: if you want to stay logged in to this other Atlassian account (because you're accessing other products with it), you can try an Incognito/Private window to log in to the first account.

Otherwise, in order to switch between accounts, you'll need to log out from one and log in to another. On the page https://id.atlassian.com/manage-profile/profile-and-visibility select your avatar (bottom left corner) > Log out

Kind regards,
Theodora