Unable to Run Whitesource/Mend Scan on Docker Container Images

[zholland]

When running a Bitbucket pipeline that kicks off a shell script to perform a Whitesource scan on our docker container images, I get the following error:

------------------------------------------------------------------------
-------------------- Start: Docker Image -------------------------------
------------------------------------------------------------------------
[INFO] [2024-02-18 06:43:01,396 +0000] - Extracting file /tmp/ws-ua_20240218064239_OMRSBL/Docker_CQUMNF/20240218064240/AWS_ID.dkr.ecr.us-east-1.amazonaws.com_maprisk_batch.tar - Size 1393349632 Bytes (1328 MBs)- Free Space 243881906176 Bytes (232583 MBs)
[INFO] [2024-02-18 06:43:06,117 +0000] - AbstractLayerScanner - scanLayersInManifest - START
[WARN] [2024-02-18 06:43:06,118 +0000] - Error extracting file 1a102d1cac2bdae8a0160ac4365d4f8653e9d6da56c793a665d556ae07fb7f82: null
[INFO] [2024-02-18 06:43:06,118 +0000] - Did not extract file 1a102d1cac2bdae8a0160ac4365d4f8653e9d6da56c793a665d556ae07fb7f82
[WARN] [2024-02-18 06:43:06,119 +0000] - Error extracting file 5f5583ba18431daa0fda68fe86f709f7196ae43cfca6aaafeba2e61e4757fb1c: null
[INFO] [2024-02-18 06:43:06,119 +0000] - Did not extract file 5f5583ba18431daa0fda68fe86f709f7196ae43cfca6aaafeba2e61e4757fb1c
[WARN] [2024-02-18 06:43:06,120 +0000] - Error extracting file 18f03fd9d7030dcbe3f7bef52d06b230767238d70b21bd5652bd6b081c88e72f: null
[INFO] [2024-02-18 06:43:06,120 +0000] - Did not extract file 18f03fd9d7030dcbe3f7bef52d06b230767238d70b21bd5652bd6b081c88e72f
[WARN] [2024-02-18 06:43:06,121 +0000] - Error extracting file 6f168b15d67585dd5d5569ed24ed1718a717f1ba60491cc02e491ce885282568: null
[INFO] [2024-02-18 06:43:06,121 +0000] - Did not extract file 6f168b15d67585dd5d5569ed24ed1718a717f1ba60491cc02e491ce885282568
[WARN] [2024-02-18 06:43:06,122 +0000] - Error extracting file 5ec1739aa6e03323ed5d4b6f7ace235469e4388fa73ba03a921322185732fb3e: null
[INFO] [2024-02-18 06:43:06,122 +0000] - Did not extract file 5ec1739aa6e03323ed5d4b6f7ace235469e4388fa73ba03a921322185732fb3e
[INFO] [2024-02-18 06:43:06,122 +0000] - AbstractLayerScanner - scanLayersInManifest - END
[INFO] [2024-02-18 06:43:06,685 +0000] - 
------------------------------------------------------------------------
-------------------- End: Docker Image ---------------------------------
------------------------------------------------------------------------

This is a new error that I started getting approximately 3 weeks ago, and prior to that it worked just fine. Nothing has changed with our docker containers, nor with our ECR/AWS permissions.

For further context, we are able to successfully pull the docker image, download the wss-unified-agent, etc. I have already reached out to Mend, and there is no issue on their end to explain this error.

We can even run Whitesource scans on our code dependencies successfully via pipelines but when scanning Docker images, it fails. I can also use Mend CLI to run a scan on our Docker Image Containers successfully(using Mend CLI is currently not a viable solution at this time).

This leads me to believe that there is an issue on the Bitbucket side that is giving me this error.

I can provide further information if need be. Thanks.

[zholland · Accepted solution]

Mend released a new wss-unified-agent version (v24.2.2). 

https://docs.mend.io/bundle/sca_user_guide/page/mend_server_release_notes.html

Running our Bitbucket pipelines with this new version has resolved the issue and completes successfully.

[bishan.vithanage]

Thanks for the update, @zholland 

I can confirm that the new version has resolved the issue. Btw, it's taking a lot longer to complete the scan than before. Are you also experiencing the same thing?

[zholland]

Didn't really notice the time difference, could be slightly longer but we just needed the results to show up again.

[Patrik S]

Hello @zholland ,

and welcome to the Community!

Based on the description of the issue, it may be the case your build has been affected by the recent docker upgrade that happened on pipelines starting on 2 Feb 2024, where we upgraded Docker Engine in pipelines from 20.10.24 to 25.0.2 to ensure we maintain our high security standards.

This change affected a small portion of customers' builds, and our engineering team published the following article with possible solutions to make your builds work in the newer version : 

• Recent docker service upgrade - Pipelines 

I would suggest going through the solutions discussed in that article and if you are still facing issues after trying those suggestions, please let us know.

Thank you, @zholland !

Patrik S

[bishan.vithanage]

Hi @Patrik S 

I have the same issue and tried DOCKER_BUILDKIT=0, but not worked for me. Any other options?

Regards,

Bishan

[zholland]

@Patrik S 

None of the suggestions seem to apply to our particular issue nor have worked. We have also tried to use our own internal builder/runner, which uses an older version of Docker (v20.10.24) and we get the same error.

Reason we tried using an older version of Docker was because it would appear that Docker engine 25 includes a breaking change to the tarball image extract function.

[Patrik S]

Thank you for sharing additional information.

Would it be possible for you to try running a previously successful pipeline and check if it also now fails?

It would be great if you could test rerunning a previously successful build using both the runner with docker engine 25 and also the runner with v20.10.24, so we can narrow down if the issue is possibly related to a docker change or not. 

[bishan.vithanage]

@Patrik S 

When I was unable to Run Mend Scan with bitbucket pipeline, I did create a .sh file contains docker login, docker pull commands and then command to execute the wss-unified-agent.jar. Then installed the latest Docker Desktop version (4.28.0) which has Docker Engine v25.0.3 and executed .sh file locally. Result was same as the bitbucket pipeline. Got the errors with image extraction.

Then I installed the Docker Desktop version 4.19.0 which has Docker Engine v23.0.5 and executed .sh file locally. Then I got the expected result. Image extraction happened without any issues.

[zholland]

@Patrik S 

Previous working Bitbucket pipeline scans that did work encounters the same error now

- unified agent v23.12.2

- docker engine v20.10.24

[Patrik S]

Hello @zholland ,

Thank you for testing and re-running the previous builds.

To further investigate this case, I've raised a support ticket for you using the email address of your community account. You should be receiving the link to the ticket soon, but if you don't, please let me know and I can share it with you here (only you and Atlassian staff can see the ticket).

To speed up the investigation, we kindly ask you to share in the support ticket the link to both the pipelines you have re-run, so we can investigate the logs.

Thank you, @zholland !

Patrik S

[zholland]

<removed>