Pipelines: Pulling docker images from a Amazon ECR repository

For a build pipeline I would like to start from a docker image stored in a Amazon EC2 Container Registry (ECR) repository. Pulling this image would normally require me to do a 'aws ecr get-login' first, but I fail to see how this would work with a in a bitbucket-pipelines.yml file. How do I get this working,? Since the pull is the first step of the build pipeline, I don't think I can execute a script?

 

Thx

4 answers

2 votes

It looks like Amazon ECR currently only allows you to generate Docker credentials that are valid for 12 hours: http://docs.aws.amazon.com/cli/latest/reference/ecr/get-authorization-token.html

Generating these for your build should work, however I suppose it is quite inconvenient to regenerate them all of the time. 

I've just outlined a potential workaround, only to stumble over the apparent lack of an actual Bitbucket Pipelines API - presumably this is just not officially documented yet (the GUI needs to manage environment variables too after all)? In other words, is a documented and supported Bitbucket Pipelines API in the pipeline already? wink

2 votes

As already mentioned by Jeroen (+1), the required Amazon ECR credentials are valid for 12 hours only, see Registry Authentication for details:

[...] you must authenticate your Docker client another way so that Amazon ECR knows who is requesting to push or pull an image. If you are using the Docker CLI, then use the docker login command to authenticate to an Amazon ECR registry with an authorization token that is provided by Amazon ECR and is valid for 12 hours. The GetAuthorizationToken API operation provides a base64-encoded authorization token that contains a user name (AWS) and a password that you can decode and use in a docker login command. 

Workaround for Bamboo

We have been facing a similar limitation when using the built-in Bamboo Docker task to push images to an ECR repository for subsequent use in the Amazon ECS tasks of our Tasks for AWS (Bamboo) add-on. In order to work around it, we have implemented an Amazon ECR Credentials Variables task, which provides the following variables for use with the Docker task (see How to push a Docker image to a repository in your Amazon ECR registry with the Bamboo Docker task for a step-by-step guide):

${bamboo.custom.aws.ecr.proxyEndpointDomain}
${bamboo.custom.aws.ecr.username}
${bamboo.custom.aws.ecr.password}

Partial workaround for Bitbucket Pipelines

Conceptually you should be able to implement a similar approach with Bitbucket Pipelines based on section Images hosted outside Docker Hub within Use Docker images as build environments in Bitbucket Pipelines, i.e. you must provide the registry details, for example:

image:
  name: $ECRPROXYENDPOINTDOMAIN/namespace/java:8u66
  username: $USERNAME
  password: $PASSWORD
  email: $EMAIL

That is, you simply provide the required credentials as Environment variables in Bitbucket Pipelines (note that $EMAIL can be arbitrary as it is ignored by Amazon ECR).

The final piece for rendering this a halfway decent workaround would be an API to update those environment variables programmatically from a scheduled external task (e.g. via a AWS Lambda function triggered by a scheduled CloudWatch event).

Unfortunately, and much to my surprise, Bitbucket Pipelines doesn't seem to offer any API yet, despite one of its main value propositions rightfully being 'configuration as code'. Thorough 'configuration as code' would also need to cover all aspects of Bitbucket Pipelines itself though, most importantly those environment variables for service discovery and credentials management etc. - a resp. feature request seems to be in order ...

Thanks for this extensive answer @Steffen Opel [Utoolity]! There is indeed a REST endpoint available to update a variable (so you can use it to update the ECR credentials): https://developer.atlassian.com/bitbucket/api/2/reference/resource/teams/%7Busername%7D/pipelines_config/variables/%7Bvariable_uuid%7D 

Late answer but might help those googler's

You can do the below and it will log you in using your AWS access & secret keys in one line.

eval $(aws ecr get-login --no-include-email --region eu-west-2)

0 vote

There is now an easier way to do this!

See Images hosted by the AWS ECR in the docs

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Posted Jun 12, 2018 in Bitbucket

Do you use any Atlassian products for your personal projects?

After spinning my wheels trying to get organized enough to write a book for National Novel Writing Month (NaNoWriMo) I took my affinity for Atlassian products from my work life and decided to tr...

23,666 views 26 12
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you