Intermittent container not found error in bitbucket runner

[Niklas Salarp]

Hi

When running a self hosted bitbucket runner we intermittently get these errors in different stages:

Status 404: {"message":"No such container: ..."}

In this case it was in a single step using just the ssh-run image but sometimes we get the same error for other images. It's also happening on different runners.

Updating the runners often but the same errors still occurs in maybe 1 of 5 runs.

Runner version:

    current: 1.413

ssh-run:0.4.1

BR

Niklas 

[Theodora Boudale]

Hi Niklas and welcome to the community!

1) Since you are using a pipe, I assume that the runner is a Linux Docker runner, is that correct?

2) Are you using one of the supported platforms for the host machine (as mentioned here)?

Linux with x64 architecture and a Linux kernel v4.0.0+. The runners have been tested on the following Linux distributions:

• Ubuntu 22.04
• Debian 11
• Centos 7
• Fedora 36
• Oracle Linux 8.6
• Amazon Linux 2

3) Are the minimum requirements met for this machine?

Minimum Requirements

Linux Docker

• A 64-Bit Linux instance with at least 8GB of RAM as a host for the runner.
  • More RAM may be required for builds with 2x-Steps or build services.
• Docker v19.03 and above

Kind regards,
Theodora

[Niklas Salarp]

Hi Theodora and thank you for both the welcome and reply!

The platform is Ubuntu 22.04.1 LTS x86_64 with more than 200GB ram.

Docker version 20.10.21.

As previously stated it works most of the time but sometimes one step fails with the described error and we must retry that step.

BR
Niklas

[Theodora Boudale]

Hi Niklas,

Thank you for the info! The next step would be to check the runner log for any additional errors. The log for Linux Docker Runners is located at

<runner_working_directory>/<runnerUuid>/runner.log

If you haven't changed the preconfigured command that starts the runner, the working directory should be /tmp and the log should be at

/tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

b609961f-891e-c872-c36b-f3f2c315d186 is the UUID of the runner, this should have the value of your own runner's UUID

If you check for entries at the date and time that particular step failed, are there any additional error messages (other than "No such container") that could give an indication as to what is going wrong?

I just noticed that your workspace is on a paid billing plan, so you can also create a ticket with the support team for further investigation. If we have an open support ticket, the engineer working on your ticket will be able to access additional logs for that step (that I am not able to access without a ticket), so the investigation should be faster. It would be helpful to provide the following info in the ticket:

• The URL of a step that fails with this error (before you rerun it)
• The runner log
• Details about the minimum requirements that are met for your platform

You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

If you have any questions, please feel free to let me know!

Kind regards,
Theodora

[Justin Beard]

Was there ever a solution to this problem?  I've recently deployed the autoscaler runners and I'm seeing the same problem.  Most of the time they work, but occasionally we'll get

Status 404: {"message":"No such container: <runnerid_stepid>_system_auth-proxy"}

The runner.log just shows the container as no longer available

[2024-03-26 22:12:57,079] Inspecting container (id: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy).
[2024-03-26 22:12:57,083] An error occurred whilst inspecting container (id: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy).

com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such container: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy"}
at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:97)
at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:32)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:140)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
[2024-03-26 22:12:57,083] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_build)
[2024-03-26 22:12:57,084] Not uploading caches. (numberOfCaches: 0, resultOrError: ERROR)
[2024-03-26 22:12:57,085] Not uploading artifacts. (numberOfArtifacts: 1, resultOrError: ERROR)
[2024-03-26 22:12:57,085] Updating step progress to PARSING_TEST_RESULTS.
[2024-03-26 22:12:57,186] Test report processing complete.
[2024-03-26 22:12:57,186] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_clone)
[2024-03-26 22:12:57,253] Container removed (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_clone)
[2024-03-26 22:12:57,253] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_clone)
[2024-03-26 22:12:57,254] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_build)
[2024-03-26 22:12:57,255] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy)
[2024-03-26 22:12:57,256] Removing container (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_pause)
[2024-03-26 22:12:57,307] Appending log line to main log.
[2024-03-26 22:13:02,225] Container removed (name: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_pause)
[2024-03-26 22:13:02,225] Updating step progress to COMPLETING_LOGS.
[2024-03-26 22:13:02,340] Shutting down log uploader.
[2024-03-26 22:13:02,340] Tearing down directories.
[2024-03-26 22:13:02,412] Cancelling timeout
[2024-03-26 22:13:02,413] Completing step with result Result{status=ERROR, error=Some(Error{key='runner.bitbucket-pipelines.build-container-failure', message='Status 404: {"message":"No such container: 1ca2faf9-cb4b-533d-8196-85c616d6c111_42a25bbe-66e1-42d7-8dcd-7eeb2b7b2da7_system_auth-proxy"}', arguments={}})}.

 The instance with the runner has 4CPU, 32GB of RAM and 150GB of diskspace.

[compumike]

@Justin Beardno solution -- we have similar problems, and I just posted at https://community.atlassian.com/t5/Bitbucket-Pipelines-Runners/Intermittent-quot-System-error-quot-quot-Status-404-quot-quot-No/qaq-p/2679929 . Hopefully the Bitbucket engineers are able to find a fix as it's quite likely a bug in the runner code.