Issues running Windows Docker Container on Pipelines Self hosted Windows Runner

Gajesh Bhat

Hello Atlassian Community,

I am facing an issue when trying to run a Windows Docker container inside a Bitbucket Self hosted Windows Runner. The problem is that the Docker container gets stuck while executing the docker run command, and after some time, the runner goes into an unhealthy state. I need to restart the runner to recover from this situation. It is strange because it's only happening with this particular container, I have similar workflows for other Windows Based Containers for Building Python GUI applications and they work smoothly with Bitbucket Pipeline Windows Runner.

Here are the details of my setup:

Pipeline Configuration: (It looks something like this)

clone: 
  lfs: true 
pipelines: 
   pull-requests: '**': 
      - step: 
          name: "Build a Rust package using Maturin on Windows Server 2022 VM" 
          runs-on: 
               - 'self.hosted' 
               - 'windows' 
                - 'server.2022' 
          script: - $env:DOCKER_HUB_PASSWORD | docker login -u $env:DOCKER_HUB_USERNAME --password-stdin 
                  - docker ps 
                  - docker images 
                  - docker pull private-registry/repo:rust_python_win_2022_container 
                  - > 
                      docker 
                      run 
                      -e ENV_ONE="$env:ENV_ONE" 
                      -e ENV_TWO="$env:ENV_TWO" 
                      -v ${PWD}:/internal_dir_where_entrypoint.ps1 
                      private-registry/repo:rust_python_win_2022_container 
          after-script: 
                      - docker logout 
                      - docker system prune -f 
                      - docker container prune -f 
                      - docker image prune -f 
                      - docker volume prune -f 
                      - docker network prune -f 

The Docker container is a Windows container with a mixed Python & Rust environment.

My questions are:

1. How can I prevent the runner from going into an unhealthy state on Windows?
2. Is there a timeline for Windows-based Docker containers to be supported on Bitbucket Pipelines?
3. What are some suggested alternatives to run Windows Docker containers on Bitbucket Pipelines? (We have already considered Bamboo, but it is out of our budget. We can consider Jenkins, but it does not have very good end-to-end integrations like Bitbucket Pipelines).

Additionally, why does Bitbucket Pipelines freeze and cause the runner to go into an unhealthy state?

FYI : I had a similar issue on the Self hosted Linux Shell Runner trying to run Rust based Docker container and it went into an unhealthy state after finishing running the pipeline. I had to then manually delete the log files in the temp directory and restart the runner to get it back up again. My best guess based on reading the logs is that this happens due to permission issue as Linux Shell docker runner does not run with super user privileges and isn't in the same group as the docker installation on the Linux machine. But I doubt this might be the case on the Windows Runner as it runs in the Administrative PowerShell by default.

Any help or hint is appreciated.

Thank you for your assistance!

Best regards,
Gajesh

Theodora Boudale

Hi Gajesh,

At the end of each step in a pipeline, the runner cleans up build directories before running the next step. If the runner can’t perform this cleanup, it will enter an Unhealthy state. Some possible causes of this issue are:

• The permissions for one or more files (or directories) within a build directory are preventing the runner from deleting files.
• A program or service is accessing one or more files (or directories) in the build directory at the time of cleanup.

Reference:

• https://confluence.atlassian.com/bbkb/troubleshooting-runners-1167819666.html

Does your Windows runner log still have entries from the last time this issue occurred? If so, could you please post here the entries around the time this issue occurred? Please make sure to sanitize any private/sensitive data prior to sharing.

The runner's log is located at

C:\Users\Username>\atlassian-bitbucket-pipelines-runner\temp\b609961f-891e-c872-c36b-f3f2c315d186\runner.log

where
C:\Users\Username>\ replace with the directory where you put the runner directory
b609961f-891e-c872-c36b-f3f2c315d186 replace with the UUID of your own runner

I also suggest debugging this specific build with the container that gets stuck outside of Pipelines, on PowerShell (on the same Windows machine where the runner is running). This is to understand if the issue is specific to the runner or not.

This is a pull-requests build, so you can do the following:

1. Clone the repo
2. Check out and pull the source and the destination branches of the PR
3. If there have been additional commits on either of these branches since the last build ran, reset each branch to the commit where the failed build ran (you can find this info in the Build log > Build setup section)
4. Merge the destination branch into the source branch
5. Then, run the commands of your yml file's script
   
   If your build uses Pipelines variables, these won't be available locally, so you'll need to define them locally before using them

Kind regards,
Theodora