Randomly getting error ' An error occurred whilst downloading data from media.'

Question

We are running bitbucket runners on kubernetes. Sometime pipelines are unable to download/upload maven cache, due to which build times increases significantly. [2024-05-20 08:47:11,877] Updating runner state to "ONLINE". [2024-05-20 08:47:41,877] Updating runner state to "ONLINE". [2024-05-20 08:47:47,991] [fc3d7b23-2, L:/10.2.35.103:57414 - R:api.media.atlassian.com/104.192.140.17:443] The connection observed an error io.netty.handler.timeout.ReadTimeoutException: null [2024-05-20 08:47:47,993] An error occurred whilst downloading data from media. io.netty.handler.timeout.ReadTimeoutException: null Wrapped by: org.springframework.web.reactive.function.client.WebClientRequestException: nested exception is io.netty.handler.timeout.ReadTimeoutException at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): *__checkpoint ⇢ Request to HEAD https://api.media.atlassian.com/file/a178cb43-88bf-4242-ad22-3a4996301363/binary?client=78604cd7-1ba1-4f01-9aa0-4ce7ae14fb4f&token=[DefaultWebClient] Original Stack Trace: at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55) at reactor.core.publisher.Mono.subscribe(Mono.java:4490) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93) at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204) at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ======================= 2024-05-16 10:45:21,829] Updating runner state to "ONLINE". [2024-05-16 10:45:33,797] [d16c8490-3, L:/10.2.26.139:53722 - R:api.media.atlassian.com/104.192.140.15:443] The connection observed an error io.netty.handler.timeout.ReadTimeoutException: null [2024-05-16 10:45:33,798] An error occurred whilst uploading data to media. io.netty.handler.timeout.ReadTimeoutException: null Wrapped by: org.springframework.web.reactive.function.client.WebClientRequestException: nested exception is io.netty.handler.timeout.ReadTimeoutException at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): *__checkpoint ⇢ Request to PUT https://api.media.atlassian.com/upload/ae79e48a-2152-44f7-a311-6958c0560585/binary?client=78604cd7-1ba1-4f01-9aa0-4ce7ae14fb4f&token= [DefaultWebClient] Original Stack Trace: at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55) at reactor.core.publisher.Mono.subscribe(Mono.java:4490) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:22

Patrik S · Answer

Hey @Rishabh ,

thank you for the confirmation.

I can see your workspace has an active Premium Bitbucket subscription that is entitled to our premium support. I went ahead and raised a ticket for your issue, and you should be receiving the ticket link in your email soon.

To speed up the investigation, as soon as you receive the email, share in the ticket the link to an example build where the cache upload/download failed, and also attach the corresponding runner.log so our support engineers can investigate further.

Thank you, @Rishabh !
Patrik S

Rishabh_9xiyyb · Answer

Hi @Patrik S ,

Thank you for the response. This connection timeout is intermittent. This runner is hosted on EKS in AWS and we do not have any firewall.

See the below snippet, it had error in maven cache but sonar cache was downloaded successfully.

Cache "maven: ~/.m2/repository": Downloading

Cache "maven: ~/.m2/repository": Error downloading. Please contact support if this error persists.

Cache "sonar: ~/.sonar": Downloading

Cache "sonar: ~/.sonar": Downloaded 291.2 MiB in 76 seconds

Cache "sonar: ~/.sonar": Extracting

Cache "sonar: ~/.sonar": Extracted in 1 seconds

Regards,

Rishabh

Patrik S · Answer

Hey @Rishabh ,

and welcome to the Community!

Based on the error message from the runner logs it looks like a connection timeout from the runner host to Bitbucket infrastructure. This may be caused by some firewall blocking that connection.

In that case, could you please confirm if all the IP addresses listed in the article below are allowed in any firewall/proxy you may have in front of the runner's host?

* IP addresses to allowlist in your corporate firewall

Those are the IP ranges used by Bitbuckets's infrastructure to which the runner will try to connect while running the pipeline builds.

Thank you, @Rishabh !

Patrik S