AWS "The provided image is invalid." when using an image built from BitBucket Pipelines

[Garth Wood]

Hi

I am trying to run a NextJS application in an AWS lambda using a docker image built from BitBucket Pipelines.

I have setup a standard lambda with 1024 memory and default storage and if I build the docker image from my local machine and push it to AWS ECR it works fine, but when I build it from BitBucket Pipelines the lambda dashboard says "Failed to update the function <my function name>: The provided image is invalid." with no additional detail.

My first assumption is BitBucket Pipelines is doing something which is causing it to fail but I'm not sure where to start looking. I also deploy other lambdas from BitBucket Pipelines (for Go etc) and it works fine.

I have tried with my app and the example from the lambda web adaptor repo and get the same result.
https://github.com/awslabs/aws-lambda-web-adapter/tree/main/examples/nextjs

As this is only an issue when building from Pipelines I imagine either Pipelines is doing something different or there is something that I am missing.

Where could I start in trying to find the issue?

[Garth Wood]

After some digging it appears that the issue lies when copying the lambda adapter binary from the remote image.

COPY --from=public.ecr.aws/awsguru/aws-lambda-adapter:0.7.0 /lambda-adapter /opt/extensions/lambda-adapter

If that binary exists locally and is copied across in the Dockerfile using "COPY ./lambda-adapter /opt/extensions/lambda-adapter" everything works. And as mentioned in my original post this only happens when building from Pipelines.

Could it be a docker version issue or something in Pipelines?

[Garth Wood]

And after that discovery it appears all that was needed was to copy using a chown. But why this is only needed in Pipelines I'm not sure.

COPY --chown=root:root --from=public.ecr.aws/awsguru/aws-lambda-adapter:0.7.0 /lambda-adapter /opt/extensions/lambda-adapter

[Igor Stoyanov]

@Garth Wood  hi. Thanks for question.

Some of the example of how to build (and deploy) docker image inside a pipeline:

step:
  script:
    # build the image
    - docker build -t $IMAGE_NAME:$IMAGE_VERSION -t $IMAGE_NAME:latest .
    - docker save --output my-docker-image.tar.gz $IMAGE_NAME
  services:
    - docker
  artifacts:
    - my-docker-image.tar.gz
step:
  script:
    # load previously saved image that will available as an artifact
    docker load --input docker-image.tar.gz
    # use the pipe to push the image to AWS ECR
    - pipe: atlassian/aws-ecr-push-image:2.0.0
      variables:
        AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID_1
        AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY_1
        AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION_1
        IMAGE_NAME: $IMAGE_NAME
  services:
    - docker

Also you could try to use Bitbucket Pipe for AWS Lambda deploy:

script:
  - pipe: atlassian/aws-lambda-deploy:1.8.2
    variables:
      FUNCTION_NAME: 'my-lambda-function'
      COMMAND: 'update'
      IMAGE_URI: '${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/my-lambda-ecr-repository:image-tag'

 Regards, Igor

[Garth Wood]

Hi Igor

Thanks for the quick response.

I'm not having issue any building the docker or pushing it to ECR, it's only when I try to select it in the lambda dashboard that I get the error.

[Igor Stoyanov]

Yes, and i suggested you to update your lambda with aws-lambda-deploy pipe using image you pushed to ecr.
I don't know the reason why do you have this issue.
I provide you a workaround of how you could update your lambda function. 

script:
  - pipe: atlassian/aws-lambda-deploy:1.8.2
    variables:
      FUNCTION_NAME: 'my-lambda-function'
      COMMAND: 'update'
      IMAGE_URI: 'your ecr docker image uri'

Regards, Igor

[Garth Wood]

The root of the issue is that AWS lambda cannot use the docker image when built from Pipelines. It doesn't matter if Pipelines tries to update the function or I try to select the image manually in the lambda dashboard.

It seems as though there is something in the Pipelines docker build process which is affecting it but I'm not sure what. Possibly some permission issue.

[Igor Stoyanov]

@Garth Wood . Working with Lambda container images :

To create a Lambda function from a container image, build your image locally and upload it to an Amazon Elastic Container Registry (Amazon ECR) repository. Then, specify the repository URI when you create the function.

Regards, Igor

[Garth Wood]

Hi Igor

I managed to figure out what it was. I posted some followup comments to this thread regarding having to use chown in the Dockerfile during the copy. This is only needed in Pipelines for some reason.

Is this something you or the dev team can investigate?

[Igor Stoyanov]

@Garth Wood . I mentioned you the process above (with pushing to ECR), because i can't see that aws-lambda-deploy pipe supports local Docker images, only ECR URI.

Regards, Igor

[Igor Stoyanov]

@Garth Wood . You provided in your example:

    - pipe: atlassian/aws-ecr-push-image:2.0.0
      variables:
        ...
        IMAGE_NAME: $IMAGE_NAME 

but IMAGE_NAME is not presented in list of variables that this pipe supports.

[Garth Wood]

I am using ECR. Please read my followup comments. There is no issue in pushing the image or with the pipes.

The issue happens in the Dockerfile when performing a COPY from a remote image, i.e.

COPY -from=public.ecr.aws/awsguru/aws-lambda-adapter:0.7.0 /lambda-adapter /opt/extensions/lambda-adapter

In Pipelines there is some ownership issue or something else I'm not aware of because it only works if I use chown, i.e

COPY --chown=root:root --from=public.ecr.aws/awsguru/aws-lambda-adapter:0.7.0 /lambda-adapter /opt/extensions/lambda-adapter

 And having to use chown is only needed in Pipelines.

Edit: just to clarify, when I say works I mean it works in AWS lambda. Without chown it still builds correctly but because of the file ownership issues and because that file being copied is an extension AWS lambda fails.

[Igor Stoyanov]

@Garth Wood . Let's summarize.
Your initial issue: Failed to update the function <my function name>: The provided image is invalid.

Your example:

    - pipe: atlassian/aws-ecr-push-image:2.0.0
      variables:
        ...
        IMAGE_NAME: $IMAGE_NAME 

What i told you:
IMAGE_NAME is not presented in list of variables that this pipe supports.

What i suggest you:
update your lambda with aws-lambda-deploy pipe using image you pushed to ecr.

script:
  - pipe: atlassian/aws-lambda-deploy:1.8.2
    variables:
      FUNCTION_NAME: 'my-lambda-function'
      COMMAND: 'update'
      IMAGE_URI: 'public.ecr.aws/awsguru/aws-lambda-adapter:0.7.0'

 Is this works?

Answering your comments:
The issue happens in the Dockerfile when performing a COPY from a remote image, i.e.
Why are you trying to copy your ECR Docker image? You could use the URI without this.

Regards, Igor

[Garth Wood]

That error message is from the AWS lambda dashboard not from Pipelines. It happens after I have already successfully uploaded the image to ECR. As mentioned previously there is no issue when uploading to ECR, whether I use a docker push or a pipe. There was only an issue when AWS lambda tried to use the image.

I'm not copying my image. The COPY is pulling in an extension from the lambda web adapter image.

https://github.com/awslabs/aws-lambda-web-adapter/tree/main

[Igor Stoyanov]

@Garth Wood . Sorry for misunderstanding. I thought you have problems with a pipe usage, not with pipelines.
Let's wait if someone from pipelines team will provide a response here.

Regards, Igor