Intermittent container not found error in bitbucket runner

Question

Hi When running a self hosted bitbucket runner we intermittently get these errors in different stages: Status 404: {"message":"No such container: ..."} In this case it was in a single step using just the ssh-run image but sometimes we get the same error for other images. It's also happening on different runners. Updating the runners often but the same errors still occurs in maybe 1 of 5 runs. current: 1.413
Runner version: ssh-run:0.4.1 BR Niklas

compumike · Answer

@Justin Beardno solution -- we have similar problems, and I just posted at https://community.atlassian.com/t5/Bitbucket-Pipelines-Runners/Intermittent-quot-System-error-quot-quot-Status-404-quot-quot-No/qaq-p/2679929 . Hopefully the Bitbucket engineers are able to find a fix as it's quite likely a bug in the runner code.

Justin Beard_lmr4fv · Answer

Was there ever a solution to this problem?  I've recently deployed the autoscaler runners and I'm seeing the same problem.  Most of the time they work, but occasionally we'll get

Status 404: {"message":"No such container: <runnerid_stepid>_system_auth-proxy"}

The runner.log just shows the container as no longer available

[2024-03-26 22:12:57,079] Inspecting container (id: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy).
[2024-03-26 22:12:57,083] An error occurred whilst inspecting container (id: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy).

com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such container: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy"}
at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:97)
at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:32)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:140)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
[2024-03-26 22:12:57,083] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_build)
[2024-03-26 22:12:57,084] Not uploading caches. (numberOfCaches: 0, resultOrError: ERROR)
[2024-03-26 22:12:57,085] Not uploading artifacts. (numberOfArtifacts: 1, resultOrError: ERROR)
[2024-03-26 22:12:57,085] Updating step progress to PARSING_TEST_RESULTS.
[2024-03-26 22:12:57,186] Test report processing complete.
[2024-03-26 22:12:57,186] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_clone)
[2024-03-26 22:12:57,253] Container removed (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_clone)
[2024-03-26 22:12:57,253] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_clone)
[2024-03-26 22:12:57,254] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_build)
[2024-03-26 22:12:57,255] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy)
[2024-03-26 22:12:57,256] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_pause)
[2024-03-26 22:12:57,307] Appending log line to main log.
[2024-03-26 22:13:02,225] Container removed (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_pause)
[2024-03-26 22:13:02,225] Updating step progress to COMPLETING_LOGS.
[2024-03-26 22:13:02,340] Shutting down log uploader.
[2024-03-26 22:13:02,340] Tearing down directories.
[2024-03-26 22:13:02,412] Cancelling timeout
[2024-03-26 22:13:02,413] Completing step with result Result{status=ERROR, error=Some(Error{key='runner.bitbucket-pipelines.build-container-failure', message='Status 404: {"message":"No such container: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy"}', arguments={}})}.

The instance with the runner has 4CPU, 32GB of RAM and 150GB of diskspace.

Theodora Boudale · Answer

Hi Niklas, Thank you for the info! The next step would be to check the runner log for any additional errors. The log for Linux Docker Runners is located at //runner.log If you haven't changed the preconfigured command that starts the runner, the working directory should be /tmp and the log should be at /tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log b609961f-891e-c872-c36b-f3f2c315d186 is the UUID of the runner, this should have the value of your own runner's UUID If you check for entries at the date and time that particular step failed, are there any additional error messages (other than "No such container") that could give an indication as to what is going wrong? I just noticed that your workspace is on a paid billing plan, so you can also create a ticket with the support team for further investigation. If we have an open support ticket, the engineer working on your ticket will be able to access additional logs for that step (that I am not able to access without a ticket), so the investigation should be faster. It would be helpful to provide the following info in the ticket: * The URL of a step that fails with this error (before you rerun it) * The runner log * Details about the minimum requirements that are met for your platform You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. If you have any questions, please feel free to let me know! Kind regards, Theodora