memory issue: while running bitbucket self hosted pipeline runner

[ajaygodugu]

Hi Team,

While running a self-hosted runner I am getting the below error, even though I have memory space in my system and disabled swap.

before it was working fine, but suddenly stopped working since from 3 days,

Status: Downloaded newer image for docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1
+ ./entrypoint.sh
[0.005s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out-of-system resources.
# An error report file with more information is saved as:
# /opt/atlassian/pipelines/runner/hs_err_pid10.log

system resources:

1) ubuntu 20.04 LTS

2) 8gb RAM machine

Please help on this !!

[ajaygodugu · Accepted solution]

Thank you @Patrik S ,

The issue is resolved now, tested it in a new machine ( 20.04 LTS Ubuntu and 4GB RAM) before I was trying the machine had the same config but The one, which I upgraded from 18.04 LTS to 20.04 LTS (focal), not sure why it has failed due to memory issue, now it's working as expected (20.04 LTS with 4 GB Ram, without any 1x, or 2x-Steps.

thanks a lot for the support again !!

[Patrik S · Accepted solution]

Hello @ajaygodugu ,

Welcome to Atlassian Community!

From the error you reported, it seems you are using Linux Docker runners and the runner container is failing to start due to memory issues.

In this case, I would like to check the following with you : 

• Could you confirm you are executing the runner in a Supported platform, that also meets the minimum requirements for Linux Docker runners ? Please also share the output of 

  cat /etc/os-release

• Can you update the runner image in your local docker to the latest version using the command below
  

  docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

  
  and then try to Start a new runner ? 
• Could you download and update your Docker to the latest version from the Official Docker website

If after following those steps you are still facing issues when starting the Linux docker runner, if possible, please try testing starting the runner in a different machine to see if the same error is reproducible.

Let me know in case you have any questions.

Thank you, @ajaygodugu .

Patrik S

[Joris Vleminckx]

We have the same issue, after updating the runner's image from 1.133 to the latest available version (basically by pulling the latest :1 image)

+ ./entrypoint.sh
[0.004s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /opt/atlassian/pipelines/runner/hs_err_pid13.log

 
Ubuntu 20.04:

ubuntu@bitbucket-runner-0:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Docker version:

ubuntu@bitbucket-runner-0:~$ docker version
Client: Docker Engine - Community
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:54:27 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:33 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0

[Joris Vleminckx]

Ok, after updating docker-ce, our runners started working again:

It seems the latest version of docker engine (20.10.22) is needed for the 1.396 runner version; we used to have 20.10.8.

ubuntu@bitbucket-runner-2:~$ sudo docker version
Client: Docker Engine - Community
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:54:27 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.22
API version: 1.41 (minimum version 1.12)
Go version: go1.18.9
Git commit: 42c8b31
Built: Thu Dec 15 22:25:58 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0