How to run the bitbucket runner behind proxy.

[Chatree Kunjai]

Hi, I tried to start the Bitbucket runner inside the server that has limited internet access. everything must pass through the proxy. After I tried to start the Bitbucket runner container, I've found an issue on start like this.

Wrapped by: com.netflix.hystrix.exception.HystrixRuntimeException: STARGATE_BITBUCKETCI_REST_SERVICE_PUT_REPOSITORY_RUNNER_STATE timed-out and no fallback available.

How do I resolve this problem?

[Caroline R]

Hi, @Chatree Kunjai! Welcome to the community! 

By the output you have shared, it seems that runner failed to reach Atlassian public gateway. In this case, could you please check if you have whitelisted the following IPs:

• Pipeline IP list

• Atlassian IP ranges

Please let me know how it goes and don't hesitate to ask for any further information if needed.

Kind regards,
Caroline 

[Chatree Kunjai]

I add the list of an IP Address to /etc/environment and /root/.docker/config.json and now it's working. Thank you for your suggestion.

[Chatree Kunjai]

Now docker of runner is running, but the runner status doesn't go to online. I check on the log and got by this.

[2021-10-14 07:38:04,830] An error occurred whilst updating runner state to "ONLINE".
java.util.concurrent.TimeoutException: null
at com.netflix.hystrix.AbstractCommand.handleTimeoutViaFallback(AbstractCommand.java:997)
at com.netflix.hystrix.AbstractCommand.access$500(AbstractCommand.java:60)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:609)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601)
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
... 14 common frames omitted
Wrapped by: com.netflix.hystrix.exception.HystrixRuntimeException: STARGATE_BITBUCKETCI_REST_SERVICE_PUT_REPOSITORY_RUNNER_STATE timed-out and no fallback available.

[Alan Hollis]

https://jira.atlassian.com/browse/BCLOUD-21158

[Caroline R]

Hi, @Chatree Kunjai! 

I'm glad to know the docker of runner is running now. As you mentioned that the runner status is not changing to online, I believe this is behind a proxy and this is not supported by Pipelines Runners yet, but we already have a feature request to implement this:

• https://jira.atlassian.com/browse/BCLOUD-21158

As you can see on the feature request, we have a workaround that can be performed:

Include the following configuration in `~/.docker/config.json` to force the docker daemon to add the proxy configuration while initiating the docker container.

  "proxies": {
    "default": {
      "httpProxy": "http://username:password@proxy2.domain.com",
      "httpsProxy": "http://username:password@proxy2.domain.com"
    }
  }

I've linked your question to this feature. Please consider adding yourself as a watcher, this way you get updated as we make progress with this. 

If you are not familiar with our Feature Request Policy, you can read more about it here: 

• Implementation of New Features Policy

Please, feel free to share any additional questions or concerns regarding this subject. 

Kind regards,
Caroline

[Chatree Kunjai]

Hi, @Caroline R 

Thanks for your reply. I already add the json config for the docker. But still got the same result. I tried to use a workaround by add the environment variable to the runner docker below but still not working.

-e http_proxy=http://username:password@proxy2.domain.com -e https_proxy=http://username:password@proxy2.domain.com

I attach the full log of runner, hope It might be help you to investigate on that.

[2021-10-15 02:32:45,908] Updating runner state to "ONLINE".
[2021-10-15 02:33:15,909] Updating runner status to "ONLINE" and checking for new steps assigned to the runner after 0 seconds and then every 30 seconds.
[2021-10-15 02:33:15,914] Updating runner state to "ONLINE".
[2021-10-15 02:33:45,918] An error occurred whilst updating runner state to "ONLINE".
java.util.concurrent.TimeoutException: null
at com.netflix.hystrix.AbstractCommand.handleTimeoutViaFallback(AbstractCommand.java:997)
at com.netflix.hystrix.AbstractCommand.access$500(AbstractCommand.java:60)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:609)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601)
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
... 14 common frames omitted
Wrapped by: com.netflix.hystrix.exception.HystrixRuntimeException: STARGATE_BITBUCKETCI_REST_SERVICE_PUT_REPOSITORY_RUNNER_STATE timed-out and no fallback available.
at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:822)
at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:807)
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1472)
at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1397)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)
at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:142)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:1142)
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:1159)
at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

[Caroline R]

Hi, @Chatree Kunjai. 

Thanks for getting back to us. It's a bit hard to say what happened without accessing the build in question, so I have created an internal ticket for you using the email of your community account.

You should have received an email with a link to the support ticket. Just in case you haven't received it, please feel free to let me know and I can post the ticket URL here. The ticket will be visible only to you and Atlassian staff, no one else can view its contents even if they have the URL.

Please feel free to let me know if you have any questions.

Kind regards,
Caroline

[Graham Gatus]

We've recently shipped a new version of the runner that supports proxy servers - see https://support.atlassian.com/bitbucket-cloud/docs/configure-a-runner-to-use-a-proxy/ for details on how to configure the runner to use a proxy.

[adrian4ascu]

Hi Graham, thank you for your post above. I just tried using the above link to configure a runner in our Kubernetes clusters and found that proxies requiring authentication are not supported (our proxy requires auth). Is there a specific reason for this? Will authentication be added in later versions of the runner? 

Thanks,

Adrian

[Patrik S]

Hello @adrian4ascu ,

I'm afraid that currently authenticated proxies are not supported in self-hosted runners, as described in the limitations section of the following documentation : 

• https://support.atlassian.com/bitbucket-cloud/docs/configure-a-runner-to-use-a-proxy/#Limitations

However, I went ahead and created a feature request in our public issue tracker to implement that functionality, which you can access using the below link : 

• https://jira.atlassian.com/browse/BCLOUD-21930

I would suggest you to add your vote there, since this helps both developers and product managers to understand the interest. Also, make sure you add yourself as a watcher in case you want to receive first-hand updates from that ticket. Please note that all features are implemented with this policy in mind: https://confluence.atlassian.com/support/implementation-of-new-features-policy-201294576.html

Thank you, @adrian4ascu .

Kind regards,

Patrik S