Deploy to AWS cloudformation and pass stack outputs to next steps

[Neethu Philip]

Is it possible  to use Bitbucket pipes to create cloudformation stacks on AWS and pass the results for processing in next steps ? This was possible in bamboo with Tasks for AWS.

[Igor Stoyanov]

@Neethu Philip , @Steffen Opel _Utoolity_ , @Sam Ware  hi.
We released a new version 0.16.0 of the pipe, where we added support to store cloudformation outputs into artifacts.
Example:

script:
  - pipe: atlassian/aws-cloudformation-deploy:0.16.0
    variables:
      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
      AWS_DEFAULT_REGION: 'us-east-1'
      STACK_NAME: 'my-stack-name'
      TEMPLATE: 'https://s3.amazonaws.com/cfn-deploy-pipe/cfn-template.json'
      CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND']
      WAIT: 'true'
      WAIT_INTERVAL: 60
      OUTPUTS_FILENAME: 'outputs'
artifacts:
  - outputs.json

outputs.json sample data:

{

   "StackId": "arn:aws:cloudformation:us-east-1:1234567890:stack/pipes-test/927c79f0-0140-11ee-82ee-0ada86db685b",
   "Outputs": [
     {
       "OutputKey": "TableName",
       "OutputValue": "pipes-test-myDynamoDBTable",
       "Description": "Table name of the newly created DynamoDB table"
     }
  ]
}

Regards, Igor 

[Darren Mansell]

Hi @Igor Stoyanov 

Would you be able to show an example of how the OutputValue would be used in a subsequent pipe?

I'm writing the outputs as an artifact as shown above and then I need the value to pass to another pipe (atlassian/aws-cloudfront-invalidate) but it needs to be parsed e.g. 

$(cat outputs.json | jq -r '.Outputs[] | select(.OutputKey == "DistributionId") | .OutputValue')

I'm unable to run arbitrary code in the aws-cloudfront-invalidate pipe and if I do it as another command in the same step, then any env vars are lost.

Thanks, Darren

[Igor Stoyanov]

@Darren Mansell  hi. I do not understand why any env vars are lost.

Try example below and if you have issues with it, please describe them with details:

script:
  - pipe: atlassian/aws-cloudformation-deploy:0.19.0
    variables:
      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
      AWS_DEFAULT_REGION: 'us-east-1'
      STACK_NAME: 'my-stack-name'
      TEMPLATE: 'https://s3.amazonaws.com/cfn-deploy-pipe/cfn-template.json'
      CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND']
      WAIT: 'true'
      WAIT_INTERVAL: 60
      OUTPUTS_FILENAME: 'outputs'

 - DISTRIBUTION_ID=$(cat outputs.json | jq -r '.Outputs[] | select(.OutputKey == "TableName") | .OutputValue')

 - pipe: atlassian/aws-cloudfront-invalidate:0.9.0
   variables:
   AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
   AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
   AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
   DISTRIBUTION_ID: $DISTRIBUTION_ID

Regards, Igor

[Darren Mansell]

@Igor Stoyanov thank you, this did work. I didn't know it could be structured like this and I'm happily passing variables between pipes in the same step.

Cheers

Darren

[Igor Stoyanov]

@Neethu Philip hi. Thanks for your question.
Currently, this is not supported in the pipe. We will investigate this and notify you if this feature will become available.

Regards, Igor

[Sam Ware]

Is there a place to upvote the feature?

[Igor Stoyanov]

@Sam Ware hi. We added this feature as a task to our backlog, so the process when feature goes from tickets to our tasks passed. Now you should wait when this feature will become available if our team will find an option of how to implement this.

Regards, Igor

[Igor Stoyanov]

Hi @Neethu Philip, @Sam Ware .
From my current investigation, the only response we have according to create-stack, update-stack is:

{
  'StackId': 'string'
} 

Is this what you want to be passed as results into the next steps?

Regards, Igor

[Steffen Opel _Utoolity_]

Hi @Igor Stoyanov

The Tasks for AWS (Bamboo) app @Neethu Philip is referring to supports Injecting task configuration via Bamboo variables, and "variables are automatically defined from AWS resources affected by a task".

So the ask here is not about the raw API response for create-stack, update-stack, rather about the outcome of this API operation, which depends on the resources provisioned by the CloudFormation template, and specifically which Outputs are defined within that template for reuse (so this is up to the template author):

The optional Outputs section declares output values that you can import into other stacks (to create cross-stack references), return in response (to describe stack calls), or view on the AWS CloudFormation console. For example, you can output the S3 bucket name for a stack to make the bucket easier to find.

Accordingly, you would need to use the describe_stacks API to await the stack operation to succeed, and then pass on all values in the Outputs array to the next pipeline steps in a suitable fashion.

You can see an example within the Variables section of the CloudFormation task (abbreviated here) - the stack at hand provisioned an SQS queue and has outputs for the SQS queue URL and ARN:

Stack 'SampleStack-TAWS-IT230-CFN2-24' generated 2 outputs:
... SampleStack-TAWS-IT230-CFN2-24.outputs: QueueURL;QueueARN
... SampleStack-TAWS-IT230-CFN2-24.outputs.QueueURL: https://sqs.us-east-1.amazonaws.com/309600995652/Bamboo-24
... SampleStack-TAWS-IT230-CFN2-24.outputs.QueueARN: arn:aws:sqs:us-east-1:309600995652:Bamboo-24

Cheers,
Steffen