We have a few tests that use the Javascript mockFs module to mock out the filesystem in node.
For the longest time these tests worked fine. Recently however they started failing only when run inside bitbucket-pipelines.
Here's a minimal reproduction https://bitbucket.org/ezequiel-wejugo/mockfs-bb-test/pipelines/results/1
What might be happening?
I imagine it could be to do with some security upgrade or other recent change on Bitbucket's side.
Hi Eze,
Just to give you some context, Bitbucket Pipelines builds run in Docker containers. For every step of the pipeline, a Docker container starts (the build container) using the image you have specified in your bitbucket-pipelines.yml file. The repo is cloned in that container and then the commands of that step's script are executed. When the step is finished, the container gets destroyed.
We have a guide for debugging Pipelines builds locally. Since you provided a public repo to reproduce, I cloned it and followed this guide.
In Step 2, I used the image node:20-alpine in the Dockerfile (the same one as your repo's). In Step 3, I used --entrypoint=/bin/sh in the command that starts the container (instead of --entrypoint=/bin/bash). The error that you see in the pipeline also occurs when debugging this locally, when I run the npm test command. So, it doesn't look like this is an issue with Bitbucket Pipelines infrastructure.
I would suggest checking the following guide, section "Scenario 1: Builds were successful earlier, but started failing recently" for possible causes and troubleshooting steps:
Please feel free to reach out if you have any questions!
Kind regards,
Theodora
Hi Theodora, thanks for the detailed response.
I run your steps locally and in my case the test passes.
In addition I run the container inside of a GKE kubernetes cluster to eliminate the possibility that running docker on Windows might be changing the result. Here too the test passes.
Assuming npm should detect a version/integrity discrepancy through the lock file, I cannot reconcile your run failing and mine succeeding.
Here's an attempt at custom integrity checking of my node_modules which is stable across installs:
$ find node_modules/ -type f | sort | xargs cat | sha512sum
1ebdcf6f06c9fd01e2213a8da35b75425e94357db18a1cdd7693eb600edf452a59a1d90ebdd72c4a4bfe8b09d5abe00b7df01fee2b8af988867398cdbf23fdad -
Do you get the same checksum?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Actually, I just checked this myself by adding a step to my bitbucket-pipelines.yml file and indeed the node_modules in the pipeline and in my local container are the same, yet the test passes one and fails in the other.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Ezequiel,
I got the same checksum as well.
Can you please confirm if you followed the same steps I am sharing below to debug this locally? This is just to confirm that you haven't done anything differently than me.
1. Create a directory for testing
mkdir testdocker2. Clone the repo in this new directory
cd testdocker
git clone git@bitbucket.org:ezequiel-wejugo/mockfs-bb-test.git localDebugRepo3. Reset the HEAD of the repository to the same commit as the failed Pipelines build
cd localDebugRepo
git reset --hard 48d74b44. Create a Dockerfile outside the clone dir
cd ..
touch my.dockerfile5. Edit the Dockerfile and add this content:
FROM node@sha256:37750e51d61bef92165b2e29a77da4277ba0777258446b7a9c99511f119db096
WORKDIR /localDebugRepo
COPY ./localDebugRepo /localDebugRepoI am pinning the Docker image to the same sha that the Pipelines build is using.
6. Build the Docker image
docker build --memory=1g --memory-swap=1g -t myimage/test -f my.dockerfile .7. Start a container using the image built in the previous step:
docker run -it --memory=4g --memory-swap=4g --memory-swappiness=0 --cpus=4 --entrypoint=/bin/sh myimage/test8. Inside the container, execute the commands of the repo's yml file
If you rerun the last successful build in the private repo, does it still succeed or does it fail?
Is the build image SHA the same or different in the last successful build and the first build that started failing (in your private repo)? You can find the image SHA if you expand the Build setup section in the Pipelines build log.
Kind regards,
Theodora
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thanks,
That now reproduces the failing test locally. It's to do with some change in the base image vs the version of the same image tag I have in my local cache.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You are very welcome, Ezequiel!
If you want, you can pin an image by digest in your yml file to ensure that no changes can be introduced.
You can find an example here:
Kind regards,
Theodora
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.