Introducing Atlassian CloudFormation Backup Machine

[#1]

Atlassian Cloudformation Backup Machine

Atlassian Cloudformation Backup Machine (Backmac) is an AWS State Machine, Lambdas and EC2 node which allow automated backups of Cloudformation Stacks produced by Atlassian's Cloudformation templates.

The IT Operations team at Atlassian built this tool internally as a backup solution which also allows for copying backups to our DR region as required.

What does the Backup Machine do?

Backmac runs a Cloudwatch rule to trigger a Lambda that will look for all Cloudformation stacks in a region with the tag 'backmac_enabled'. These will be added to an SQS queue and the State Machine execution will be triggered.

The State Machine orchestrates a series of Lambdas and a dedicated high IOPS compute node and works through the following:

• Mounts the EFS volume on the compute node
• If needed, create or resize a backup EBS volume to copy the EFS content onto
• Trigger an RDS Snapshot
• Do a parallel rsync of shared home (on EFS) to the backup EBS volume
• Snapshot the EBS volume
• Copy the snapshots to dr region

The code

https://bitbucket.org/atlassian/aws-backup-machine/src/master/

Pre-deployment configuration

1. You will need an AWS account, with either the AWS managed "AdministratorAccess" policy or the effective "Allow *" for the following services:
   1. EC2
   2. IAM
   3. CloudWatch
   4. S3
   5. SSM
   6. Lambda
   7. Step Functions
2. Decide which AWS regions you want to create stacks in

Deployment

Backmac is deployed in a two step process. 

1. run package.sh <region> from the root of the repo
2. deploy the generated Cloudformation template .yaml file

The template creates:

1. An IAM role with all the access Backmac needs to operate
2. An EC2 node that runs the rsync process
3. A bunch of Lambdas used by the State Machine
4. A State Machine to orchestrate the Lambdas

Post-deployment configuration

Make sure at least one stack is tagged with 'backmac_enabled=true'

Backmac will run on the cron schedule entered during the creation of Backmac, and will back up any stack that is tagged as above.

Cleaning up old snapshots

Backmac will also create an additional Lambda to clean up any backups older than their 'backup_delete_after' date (which Backmac will set to 30 days after backup by default).

Support

Atlassian CloudFormation Backup Machine is not supported by Atlassian. It is provided as is.

More information

More detailed information about the Backup Machine, and migrating to AWS CloudFormation in general, is provided in this video.

[Edwin Kyalangalilwa]

Thank you!