Bitbucket pipeline failed on tests run

[Lorenzo Biotti]

For a nest.js project I have a pipeline that executes unit and e2e tests.
Locally  they run successfully.
In the pipeline, randomly some of them fail due to this error:
"A jest worker process (pid=179) was terminated by another process: signal=SIGKILL, exitCode=null. Operating system logs may contain more information on why this occurred.
at ChildProcessWorker._onExit (../node_modules/jest-worker/build/workers/ChildProcessWorker.js:370:23)"

How can I fix? Where should I check?

[bodva]

@Lorenzo Biotti 

If you are still experiencing the problem, you could review the discussion under the previous comment. 

We had the same errors as you had, and we found two ways:

• You could migrate to "swc" instead of ts-node and add the custom runtime "@side/jest-runtime".
• You could set "size: 2x" for the step when experiencing this issue.

With the first approach, you can avoid substantial memory leaks, which have a combination of node > 16.11 and jest.

[Theodora Boudale]

Hi @Lorenzo Biotti and welcome to the community!

The issue may be related to this bug report I found in the jest repository here:

• https://github.com/jestjs/jest/issues/13976

I see that the bug report has been fixed, so you could try using the latest version of jest and see if that fixes the issue.

Just a heads up, other customers have reported that Pipelines builds using the jest framework are hanging or failing because of memory issues.

You can check the following knowledge base article we have created, specifically the section Scenario 2.2: Builds using Jest Test Framework are slow or frequently hang (based on the Pipeline build minutes consumption) or failed with Container “Build” exceeded memory limit error:

• https://confluence.atlassian.com/bbkb/troubleshooting-bitbucket-pipelines-1141505226.html

The suggestions there might help as well.

Kind regards,
Theodora

[bodva]

Hey @Theodora Boudale 

Unfortunately has the same problem for jest 29.6.2. Could it be that pipelines kill the jest process?

[Theodora Boudale]

Hi @bodva,

A process may be killed if there is not enough memory. Regular pipelines steps have 4096 MB of memory in total, while large build steps (which you can define using size: 2x) have 8192 MB in total. If you have e.g. 7 jest workers running and each of them is configured to use 2 GB memory, there won't be enough memory for all.

You can check the suggestion in the knowledge base article I shared above:

You can check the following knowledge base article we have created, specifically the section Scenario 2.2: Builds using Jest Test Framework are slow or frequently hang (based on the Pipeline build minutes consumption) or failed with Container “Build” exceeded memory limit error:

• https://confluence.atlassian.com/bbkb/troubleshooting-bitbucket-pipelines-1141505226.html

If that doesn't help you can create a ticket with the support team via https://support.atlassian.com/contact/.

Kind regards,
Theodora

[bodva]

Hey @Theodora Boudale 

Thanks for the replay.

I am working with option maxWorkers = 2 to avoid the case with a memory limit.

I don't receive a memory limit problem.

My problem is the same as the topic starter has.

I checked the article you linked and found no solution to my problem. 

I am using 1x steps. And the most unexpected thing here is that sometimes it killed unit tests, sometimes e2e. And in some runs - it works. 

[Theodora Boudale]

Hi @bodva,

Even if your build doesn't fail with a memory limit error, a certain process can still be killed if memory usage is close to the limit.

I would suggest adding in your yml file the commands mentioned in the article, at the beginning of the step that has this issue.

- while true; do ps -aux && sleep 5; done &
- while true; do echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && sleep 0.1; done &

These will print in the Pipelines build log memory usage per process and total memory used throughout the build. Then, the next time you face this error, you can check in the build log memory usage before this error and if it seems to be close to the limit of 4 GB for the 1x step you are using.

If it is, then the process is most likely killed because of memory issues. You can also see which processes consume memory. You can then either configure the build to use less memory or, if this is not possible, try using a 2x step.

I just saw that another user reported the same error and they figured out the issue was a bug with jest on Node JS 16.11.0+: https://github.com/jestjs/jest/issues/11956 You can take a look in and see if you have a similar set up as the one described in the bug report.

Kind regards,
Theodora

[bodva]

Hey @Theodora Boudale 

I receive an error when trying to add the command to the step.

ps: unrecognized option: u
BusyBox v1.36.1 (2023-07-27 17:12:24 UTC) multi-call binary.
Usage: ps [-o COL1,COL2=HEADER] [-T]
Show list of processes
-o COL1,COL2=HEADER Select columns for display
-T Show threads
But I guess you are right about the memory, because adding 2x to the step fix the issue.

And thanks for the link to the issue, I checked it. And find that I could try to use custom runtime for jest:

 "runtime": "@side/jest-runtime" 

It looks like this can address the problem with memory leaks in jest with node > 16.11, until the PR with this fix would be merged into jest.

[Theodora Boudale]

Hi @bodva,

Thank you for the update, it's good to hear that the linked issue helped!

Regarding the error when adding the commands in your yml file, it sounds like the Linux distribution of the Docker image you use as a build container does not recognize the option u for the ps command. You can omit it from ps -aux, and try using ps -ax, but in this case, the percentage of memory and CPU used by each process won't be displayed.

Kind regards,
Theodora

[bodva]

Hi @Theodora Boudale 

Thanks for the suggestion. I am using 

node:18-alpine

and it looks like ps in this image does not recognize -x either. 

The same as the next line doesn't work in the alpine.

So, as I can see, this way to debug this problem can't be used for node images built from alpine.

[Theodora Boudale]

Hi @bodva,

I looked a bit more into this and you can use these commands with the alpine image if you add the package procps:

I used these commands at the beginning of a yml file's script for a step that uses node:18-alpine:

- apk add procps
- while true; do ps -aux && sleep 5; done &
- while true; do echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && sleep 0.1; done &

Is this something that works for you?

Kind regards,
Theodora

[bodva]

Hey @Theodora Boudale 

That makes pa -aux work, but then we receive the error:

cat: can't open '/sys/fs/cgroup/memory/memory.memsw.usage_in_bytes': No such file or directory

 But don't worry. I just wanted to let you know that I don't need this debug.

[Theodora Boudale]

Hi @bodva

I cannot reproduce this specific error. I understand that you don't need this right now, since the issue is resolved.

In case you ever do need it in the future, you can create a support ticket and we will look into this. If we have an open support ticket, the engineer working on it will be able to view the build logs and your setup and investigate further.

Kind regards,
Theodora

[bodva]

Hi @Theodora Boudale 

Thanks for your support.