SAM local fails to run containers inside pipeline

Question

I have updated from python3.9 runtime with to 3.11. For my pipeline I use an image based on python:3.11-alpine. I have aws-sam version 1.115.0 installed. I run integration test based on exported postman collections. The sam build inside pipeline succeeds. The sam local start api command also works. The first few requests to my lambda functions are successful. After that every request returns with 502 from the lambdas. From the pipelines docker sections I get this log, which I think is responsible for this issue:

level=error msg="AuthZRequest for GET /v1.35/distribution/public.ecr.aws/lambda/python:3.11-x86_64/json returned error: authorization denied by plugin pipelines: "

When I run the test locally everything works fine. I will really appreciate some input on solving this issue. Thanks.

viktor_varju · Answer

Turns out SAM was making request to pull the latest Lambda images from AWS public ecr. I guess there is some sort of rate limiting mechanism in Bitbucket pipelines causing only the first few pulls to succeed from ecr. I solved the issue by starting SAM with --skip-pull-image option.

Theodora Boudale · Answer

Hi Viktor and welcome to the community!

We have restricted certain options for Docker commands in builds running on Atlassian's infrastructure for security reasons. You can take a look at our documentation, the restricted commands are listed in the section "Full list of restricted commands":

* https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/#Running-Docker-commands

Based on the error message, it looks like the Docker containers are started with one of the options we do not allow.

Can you see in the build log the exact docker command that is running and giving this error? Or is there any additional output in the build log to indicate what option is causing this error?

If the docker run commands cannot be configured in order to remove the restricted option(s), you can look into running this step on your own infrastructure with one of our self-hosted runners:

* https://support.atlassian.com/bitbucket-cloud/docs/runners/

The restrictions we have only apply to the pipelines executed on our cloud infrastructure, they do not apply to self-hosted runners.

If you use a self-hosted Linux Docker runner, you will need to use a custom docker-in-docker image for the docker service so that you don't get the same error:

* https://support.atlassian.com/bitbucket-cloud/docs/configure-your-runner-in-bitbucket-pipelines-yml/#Custom-docker-in-docker-image

Please feel free to reach out if you have any questions.

Kind regards,
Theodora