Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Why are my deployment variables not propagating through my pipeline correctly?

I have a pipeline that dockerizes a Node.js application, pushes the image to AWS ECR and then deploys the image to AWS ECS. 

I am using atlassian/aws-ecr-push-image:1.2.0 to push to ECR and atlassian/aws-ecs-deploy:1.0.3 to deploy to ECS. Both of these pipes require the following AWS variables:

AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

but state in the README that they can be omitted if defined in the context.

I have defined these variables as deployment variables, which appears to work correctly for atlassian/aws-ecr-push-image:1.2.0 but not for atlassian/aws-ecs-deploy:1.0.3.

 

This fails on the ECS deploy step, stating it is missing the required AWS variables:

 - pipe: atlassian/aws-ecr-push-image:1.2.0
variables:
IMAGE_NAME: $IMAGE_NAME

TAGS: '${COMMIT_HASH} latest'

- pipe: atlassian/aws-ecs-deploy:1.0.3

variables:

CLUSTER_NAME: $ECS_CLUSTER_NAME
SERVICE_NAME: $ECS_SERVICE_NAME

TASK_DEFINITION: 'my_task_definition.json'
DEBUG: "true"
FORCE_NEW_DEPLOYMENT:"true" 

 

Even though both pipes have the exact same expected AWS variables, the ECR pipe receives the AWS variables as expected, but the ECS pipe does not. Explicitly setting the variables causes both pipes to work as expected:


- pipe: atlassian/aws-ecr-push-image:1.2.0
variables:
IMAGE_NAME: $IMAGE_NAME

TAGS: '${COMMIT_HASH} latest'

- pipe: atlassian/aws-ecs-deploy:1.0.3

variables:

AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

CLUSTER_NAME: $ECS_CLUSTER_NAME
SERVICE_NAME: $ECS_SERVICE_NAME

TASK_DEFINITION: 'my_task_definition.json'
DEBUG: "true"
FORCE_NEW_DEPLOYMENT:"true" 


Is this expected behaviour? Can the variables only be passed through one pipe from context? What am I doing wrong here?  

3 answers

1 accepted

0 votes
Answer accepted

@Shane McNamara also, try to split these executions into two steps, not one.

BEsides that, this looks like a bug inside pipeline infra, we will report this, thank you

0 votes

@Shane McNamara can you please share full traceback when the second pipe fails without passing variables explicitly?

This will help us a lot to find root cause.

Regards, Galyna

+ docker container run \
--volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \
--volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro \
--volume=/usr/local/bin/docker:/usr/local/bin/docker:ro \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/aws-ecs-deploy:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/aws-ecs-deploy \
--workdir=$(pwd) \
--label=org.bitbucket.pipelines.system=true \
--env=BITBUCKET_STEP_TRIGGERER_UUID="$BITBUCKET_STEP_TRIGGERER_UUID" \
--env=BITBUCKET_REPO_FULL_NAME="$BITBUCKET_REPO_FULL_NAME" \
--env=BITBUCKET_GIT_HTTP_ORIGIN="$BITBUCKET_GIT_HTTP_ORIGIN" \
--env=BITBUCKET_PROJECT_UUID="$BITBUCKET_PROJECT_UUID" \
--env=BITBUCKET_REPO_IS_PRIVATE="$BITBUCKET_REPO_IS_PRIVATE" \
--env=BITBUCKET_WORKSPACE="$BITBUCKET_WORKSPACE" \
--env=BITBUCKET_DEPLOYMENT_ENVIRONMENT_UUID="$BITBUCKET_DEPLOYMENT_ENVIRONMENT_UUID" \
--env=BITBUCKET_REPO_OWNER_UUID="$BITBUCKET_REPO_OWNER_UUID" \
--env=BITBUCKET_BRANCH="$BITBUCKET_BRANCH" \
--env=BITBUCKET_REPO_UUID="$BITBUCKET_REPO_UUID" \
--env=BITBUCKET_PROJECT_KEY="$BITBUCKET_PROJECT_KEY" \
--env=BITBUCKET_DEPLOYMENT_ENVIRONMENT="$BITBUCKET_DEPLOYMENT_ENVIRONMENT" \
--env=BITBUCKET_REPO_SLUG="$BITBUCKET_REPO_SLUG" \
--env=CI="$CI" \
--env=BITBUCKET_REPO_OWNER="$BITBUCKET_REPO_OWNER" \
--env=BITBUCKET_STEP_RUN_NUMBER="$BITBUCKET_STEP_RUN_NUMBER" \
--env=BITBUCKET_BUILD_NUMBER="$BITBUCKET_BUILD_NUMBER" \
--env=BITBUCKET_GIT_SSH_ORIGIN="$BITBUCKET_GIT_SSH_ORIGIN" \
--env=BITBUCKET_PIPELINE_UUID="$BITBUCKET_PIPELINE_UUID" \
--env=BITBUCKET_COMMIT="$BITBUCKET_COMMIT" \
--env=BITBUCKET_CLONE_DIR="$BITBUCKET_CLONE_DIR" \
--env=PIPELINES_JWT_TOKEN="$PIPELINES_JWT_TOKEN" \
--env=BITBUCKET_STEP_UUID="$BITBUCKET_STEP_UUID" \
--env=BITBUCKET_DOCKER_HOST_INTERNAL="$BITBUCKET_DOCKER_HOST_INTERNAL" \
--env=DOCKER_HOST="tcp://host.docker.internal:2375" \
--env=BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" \
--env=BITBUCKET_PIPE_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/aws-ecs-deploy" \
--env=CLUSTER_NAME="$ECS_CLUSTER_NAME" \
--env=DEBUG="true" \
--env=SERVICE_NAME="$ECS_SERVICE_NAME" \
--env=TASK_DEFINITION="alpha_task_definition.json" \
--add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL" \
bitbucketpipelines/aws-ecs-deploy:1.0.3
Unable to find image 'bitbucketpipelines/aws-ecs-deploy:1.0.3' locally
1.0.3: Pulling from bitbucketpipelines/aws-ecs-deploy

[Removed output from docker pull]

Status: Downloaded newer image for bitbucketpipelines/aws-ecs-deploy:1.0.3
[31m✖ Validation errors:
AWS_ACCESS_KEY_ID:
- required field
AWS_DEFAULT_REGION:
- required field
AWS_SECRET_ACCESS_KEY:
- required field

 

Notice the ENV variables seem to be missing in the docker run command. They are present in that command in the previous part

atlassian/aws-ecr-push-image:1.2.0
Like Galyna Zholtkevych likes this

@Shane McNamara yeah, I see that. Thank you for reporting this. If pipelines team consider it as a bug, it will be fixed in the nearest future and your output will help to resolve issue quickly

What name did you use for the 'deployment' label of the parent step?

 

```

 - step: 
name: Deploy to Production
deployment: Production
services: - docker
image: atlassian/pipelines-awscli
trigger: manual
script:
- pipe: atlassian/aws-ecr-push-image:1.2.1
variables:
IMAGE_NAME: $SERVICE_NAME
-
echo $TASK_DEFINITION_base64 | base64 -d > task-definition.json
-
pipe: atlassian/aws-ecs-deploy:1.3.0
variables:
CLUSTER_NAME: $AWS_CLUSTER_NAME
SERVICE_NAME: $SERVICE_NAME
TASK_DEFINITION: task-definition.json
WAIT: 'true'  

Deployment is named `alpha`

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket Pipelines

What We Learned When We Researched Open Source Vulnerabilities in 7 Popular Coding Languages

...hey are a part of us, shaping how we interact with the world around us. The same holds true for programming languages when we think about how different kinds of vulnerabilities raise their heads in t...

1,323 views 0 3
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you