[2024-11-20 12:17:20,054] Appending log line to main log.
[2024-11-20 12:17:27,577] Initiating artifact upload.
[2024-11-20 12:17:27,866] Successfully got total chunks FileChunksInfo{dataSize=262224310B, totalChunks=6}.
[2024-11-20 12:17:27,870] Uploading 6 chunks to s3
[2024-11-20 12:17:27,872] Getting s3 upload urls for artifact.
[2024-11-20 12:17:28,042] Appending log line to main log.
[2024-11-20 12:17:36,462] Updating runner state to "ONLINE".
[2024-11-20 12:17:38,864] [13d480a9-1, L:/192.168.150.119:64555 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:38,865] [f58a16bd-1, L:/192.168.150.119:64556 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:38,865] [98ad0830-1, L:/192.168.150.119:64553 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:38,865] [3688902a-1, L:/192.168.150.119:64552 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:38,865] [93d7c715-1, L:/192.168.150.119:64554 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:51,112] [6662d841-1, L:/192.168.150.119:64569 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:51,123] [c69d1a83-1, L:/192.168.150.119:64572 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:51,123] [326f0762-1, L:/192.168.150.119:64570 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:51,123] [84638be8-1, L:/192.168.150.119:64573 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:17:51,123] [a803374a-1, L:/192.168.150.119:64571 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:18:05,365] [bc13356c-1, L:/192.168.150.119:64596 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:18:05,385] [4f188350-1, L:/192.168.150.119:64598 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:18:05,385] [f67fc8d9-1, L:/192.168.150.119:64597 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:18:05,385] [ec79b859-1, L:/192.168.150.119:64599 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:18:05,385] [494c338d-1, L:/192.168.150.119:64600 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.15.179.177:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:18:06,462] Updating runner state to "ONLINE".
[2024-11-20 12:18:23,618] [57659f01-1, L:/192.168.150.119:64624 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/52.216.40.233:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-20 12:18:23,623] [0c9c6507-1, L:/192.168.150.119:64627 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/52.216.40.233:443] An exception has been observed post termination, use DEBUG level to see the full stack: io.netty.handler.timeout.ReadTimeoutException
[2024-11-20 12:18:23,623] [01623a99-1, L:/192.168.150.119:64626 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/52.216.40.233:443] An exception has been observed post termination, use DEBUG level to see the full stack: io.netty.handler.timeout.ReadTimeoutException
[2024-11-20 12:18:23,623] [0cf33d6e-1, L:/192.168.150.119:64625 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/52.216.40.233:443] An exception has been observed post termination, use DEBUG level to see the full stack: io.netty.handler.timeout.ReadTimeoutException
[2024-11-20 12:18:23,623] [3daaff0f-1, L:/192.168.150.119:64628 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/52.216.40.233:443] An exception has been observed post termination, use DEBUG level to see the full stack: io.netty.handler.timeout.ReadTimeoutException
[2024-11-20 12:18:23,623] Error while uploading file to s3
io.netty.handler.timeout.ReadTimeoutException: null
Wrapped by: org.springframework.web.reactive.function.client.WebClientRequestException: nested exception is io.netty.handler.timeout.ReadTimeoutException
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint Γçó Request to PUT https://micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/artifact/{cf74d650-59...REMOVED...97fcfc38d}/{6aaf5109...REMOVED...ca7458ffec}/{4afec5e1-...REMOVED...96acb1bc8}/artifact_{64...REMOVED...c663}.tar.gz?partNumber=5&uploadId=HUt1y...REMOVED...Dg21...REMOVED...vAQH4czg--&X-Amz-Security-Token=IQoJb3...REMOVED...3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20241120T111731Z&X-Amz-SignedHeaders=host&X-Amz-Credential=ASIATNIC...REMOVED...%2F20241120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Expires=900&X-Amz-Signature=20e...REMOVED...2d8 [DefaultWebClient]
Original Stack Trace:
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)
at reactor.core.publisher.Mono.subscribe(Mono.java:4491)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:96)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204)
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:96)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:415)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:89)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:96)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
at reactor.netty.http.client.HttpClientConnect$HttpObserver.onUncaughtException(HttpClientConnect.java:403)
at reactor.netty.ReactorNetty$CompositeConnectionObserver.onUncaughtException(ReactorNetty.java:700)
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onUncaughtException(DefaultPooledConnectionProvider.java:211)
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onUncaughtException(DefaultPooledConnectionProvider.java:464)
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:247)
at reactor.netty.channel.FluxReceive.onInboundError(FluxReceive.java:468)
at reactor.netty.channel.ChannelOperations.onInboundError(ChannelOperations.java:508)
at reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:145)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:98)
at io.netty.handler.timeout.ReadTimeoutHandler.channelIdle(ReadTimeoutHandler.java:90)
at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:525)
at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:497)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1575)
Wrapped by: com.atlassian.pipelines.runner.core.exception.S3UploadException: Failed to upload chunk, part number 5
at com.atlassian.pipelines.runner.core.util.file.upload.S3MultiPartUploaderImpl.lambda$uploadChunk$16(S3MultiPartUploaderImpl.java:167)
at io.reactivex.internal.operators.single.SingleResumeNext$ResumeMainSingleObserver.onError(SingleResumeNext.java:73)
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onError(FlowableSingleSingle.java:97)
at io.reactivex.subscribers.SerializedSubscriber.onError(SerializedSubscriber.java:142)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onError(FlowableRepeatWhen.java:112)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.checkTerminate(FlowableFlatMap.java:572)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:379)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:371)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.innerError(FlowableFlatMap.java:611)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onError(FlowableFlatMap.java:677)
at io.reactivex.internal.subscriptions.EmptySubscription.error(EmptySubscription.java:55)
at io.reactivex.internal.operators.flowable.FlowableError.subscribeActual(FlowableError.java:40)
at io.reactivex.Flowable.subscribe(Flowable.java:14935)
at io.reactivex.Flowable.subscribe(Flowable.java:14882)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmitScalar(FlowableFlatMap.java:234)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:152)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipCoordinator.drain(FlowableZip.java:249)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipSubscriber.onNext(FlowableZip.java:381)
at io.reactivex.processors.UnicastProcessor.drainFused(UnicastProcessor.java:362)
at io.reactivex.processors.UnicastProcessor.drain(UnicastProcessor.java:395)
at io.reactivex.processors.UnicastProcessor.onNext(UnicastProcessor.java:457)
at io.reactivex.processors.SerializedProcessor.onNext(SerializedProcessor.java:103)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenSourceSubscriber.again(FlowableRepeatWhen.java:171)
at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onError(FlowableRetryWhen.java:76)
at io.reactivex.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onError(SingleToFlowable.java:67)
at io.reactivex.internal.operators.single.SingleUsing$UsingSingleObserver.onError(SingleUsing.java:175)
at io.reactivex.internal.operators.single.SingleMap$MapSingleObserver.onError(SingleMap.java:69)
at io.reactivex.internal.operators.single.SingleMap$MapSingleObserver.onError(SingleMap.java:69)
at io.reactivex.internal.operators.single.SingleObserveOn$ObserveOnSingleObserver.run(SingleObserveOn.java:79)
at brave.propagation.CurrentTraceContext$1CurrentTraceContextRunnable.run(CurrentTraceContext.java:264)
at com.atlassian.pipelines.common.trace.rxjava.CopyMdcSchedulerHandler$CopyMdcRunnableAdapter.run(CopyMdcSchedulerHandler.java:74)
at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:608)
at brave.propagation.CurrentTraceContext$1CurrentTraceContextRunnable.run(CurrentTraceContext.java:264)
at com.atlassian.pipelines.common.trace.rxjava.CopyMdcSchedulerHandler$CopyMdcRunnableAdapter.run(CopyMdcSchedulerHandler.java:74)
at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1575)
[2024-11-20 12:18:23,633] Updating step progress to PARSING_TEST_RESULTS.
[2024-11-20 12:18:23,900] Test report processing complete.
[2024-11-20 12:18:23,900] Updating step progress to COMPLETING_LOGS.
[2024-11-20 12:18:24,055] Appending log line to main log.
[2024-11-20 12:18:24,180] Shutting down log uploader.
[2024-11-20 12:18:24,388] Tearing down directories.
[2024-11-20 12:18:24,752] Cancelling timeout
[2024-11-20 12:18:24,753] Completing step with result Result{status=ERROR, error=Some(Error{key='runner.artifact.upload-error', message='Error occurred whilst processing an artifact', arguments={}})}.
[2024-11-20 12:18:25,015] Setting runner state to not executing step.
[2024-11-20 12:18:25,015] Waiting for next step.
[2024-11-20 12:18:25,016] Finished executing step. StepId{accountUuid={c8a2a3b3-90b0-4278-882e-9aef13704321}, repositoryUuid={cf74d650-59f4-48ce-ac41-67897fcfc38d}, pipelineUuid={6aaf5109-f4d0-4852-92ea-79ca7458ffec}, stepUuid={4afec5e1-57ce-4017-9686-26196acb1bc8}}
[2024-11-20 12:18:36,458] Updating runner state to "ONLINE".
What I've tried:
- Creating new runners -> same issue.
- Updating Java -> same issue.
- Installing runners on another machine -> same issue.
It is easy to replicate with this pipeline: