Backup with Confluence Cloudformation Template

[Steven Yong]

Hi all,

Hopefully this is the right place to ask. So i used the Cloudformation template for Confluence Server available at https://bitbucket.org/atlassian/atlassian-aws-deployment/src/master/templates/ConfluenceServer.template.yaml but i'm wondering how is the backup configured? doesn't seem to snapshot or image the EC2. Reason why is if i run the cloudformation and it builds 1 EC2, i complete the Confluence configuration, put in my license, import my data etc etcx etc. But, if i increase the Load Scaling to say 2 instances, it builds another one but it's just a brand new EC2 with barebone Confluence. Then if i decrease it back to say 1 instance, it terminates my first EC2 with all the data and doesn't back it up and im back to a brand new Confluence application.

I guess i'm just trying to figure out how this cloudformation script keeps the data, i intially thought it would automatically snapshot the EBS and re-mount it when it builds a new instance but that's not the case.

[Adam Harvey]

So using the CloudFormation template to create Confluence, essentially you are using a "cloud application pattern" in the sense that the servers themselves have no real data.  (See Pets v Cattle: http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ )  They can be spun up, turned off, and you won't lose anything. Instead, all data is stored separately either in the RDS database (users, space lists, configuration settings, etc) or on the EFS File Share (actual attached files).  The file share is then MOUNTED to the EC2 hosts, so they have access to it.  So 1, 2, or 15 EC2 hosts can all access the same data and the same RDS database.   So a backup of the solution is to backup the RDS database, and then take a snapshot of the EFS File share.  That's essentially what the Backmac template does, which if you're using the Confluence one, you may want to look into as well. Restoration is similar, in the sense that you'd restore the RDS database and then restore the EFS snapshot and re-mount it to the EC2 hosts.

Hope this helps!

[Steven Yong]

Hi Adam,

Looking at that cloudformation script it looks like it creates a EBS volume, not a EFS and the script is ran inside the OS rather than from CloudFormation which is why im confused in terms of how data would be kept/backed up. In saying that ill take a look at Pets v Cattle link.

[Adam Harvey]

@Steven Yong Ah sorry, yes I wrote too quickly.  Yes, you can't snapshot an EFS directly.  There's a nice description in the process that Atlassian's Backmac tool uses here:

https://community.atlassian.com/t5/Data-Center-articles/Introducing-Atlassian-CloudFormation-Backup-Machine/ba-p/881556

So your central EFS which is mounted to your Confluence nodes is always there, giving data to both notes.  To backup, essentially you mount that EFS to another machine, create an EBS volume that can contain the entire EFS share, copy the data to the EBS share, and snapshot that EBS.  My team used to (and in some cases) still does this with a permanent EC2 host we use for that process, but the Backmac approach is much more cloud native and beneficial - it spins up an EC2 to be used only during the time the backup is needed, and it spins up one with sufficient disk IO to be able to do it very quickly - before tearing it all down. (saving the cost of running the EC2 when its not needed)  It also puts the RDS and EBS Backup into another region, which is what you truly need for a good backup.

So you could either use Backmac out of the book, or effectively mirror it's approach for your own needs.

[Steven Yong]

I think you are still confused with my question, i can understand how the Confluence Data Centre scripts work which seems ok to me, my question is more directed at the ConfluenceServer script are not using Data Center in our environment.

The ConfluenceServer.template.yaml script contains a section where it creates a EBS volume unlike the Data Center one which creates a EFS volume, hence my initial question in the first place. See part of the script below, true that it snapshots it and remounts it if a new EC2 is spun up, but i guess the question still remains is what happens when you want to load balance 2 or more EC2s, this means that script will spin up 2nd EC2, snapshot the EBS from 1st EC2, creates2nd EBS volume and restores it but that would mean both EC2s will contain 2 seperate EBS volumes with different data when load balanced and my initial question would be, yes there's no backup of the data here and i guess i can schedule a nightly snapshot of the EBS volume but there will be 2 or more individual EBS volumes depending on how many i want load balanced.

                  #!/usr/bin/env bash
                  # confluence server doesn't persist the home directory, so we'll do it manually via a reusable EBS

                  set -xe

                  ebsToMount=''
                  currentAz=$(curl http://instance-data/latest/meta-data/placement/availability-zone)
                  instanceId=$(curl http://instance-data/latest/meta-data/instance-id)
                  existingEbs=$(aws --region=${AWS::Region} ec2 describe-volumes --filters Name=tag:Name,Values=*local-home Name=tag:stack_id,Values=${AWS::StackId} | jq .Volumes[])
                  existingEbsVolumeId=$(echo "$existingEbs" | jq -r .VolumeId)
                  deviceName="/dev/xvdb"
                  mountPoint="/var/atlassian/application-data/confluence"

                  if [[ -n "$(echo "$existingEbs" | jq length)" ]]; then
                    if [[ "$(echo "$existingEbs" | jq -r .AvailabilityZone)" != "$currentAz" ]]; then
                      echo "Existing EBS volume isn't in our AZ; creating snapshot to restore volume in $currentAz"
                      existingEbsSnapshot=$(aws --region=${AWS::Region} ec2 create-snapshot --volume-id "$existingEbsVolumeId" --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=${AWS::StackName} local-home temp snapshot},{Key=stack_id,Value=${AWS::StackId}}]')
                      existingEbsSnapshotId=$(echo "$existingEbsSnapshot" | jq -r .SnapshotId)

                      echo "Waiting for snapshot creation to finish"
                      aws --region=${AWS::Region} ec2 wait snapshot-completed --snapshot-ids "$existingEbsSnapshotId"

                      echo "Creating new EBS volume from snapshot in $currentAz"
                      ebsToMount=$(aws --region=${AWS::Region} ec2 create-volume --snapshot-id "$existingEbsSnapshotId" --availability-zone "$currentAz" --volume-type gp2 ${encryption_option} --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=${AWS::StackName} Confluence Server local-home},{Key=stack_id,Value=${AWS::StackId}}]')
                    else
                      echo "Existing EBS volume is in our AZ; using existing EBS"
                      ebsToMount=$existingEbs
                    fi
                  else
                    echo "Creating new EBS volume in $currentAz"
                    ebsToMount=$(aws --region=${AWS::Region} ec2 create-volume --size "${LocalHomeVolumeSize}" --availability-zone "$currentAz" --volume-type gp2 ${encryption_option} --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=${AWS::StackName} Confluence Server local-home},{Key=stack_id,Value=${AWS::StackId}}]')
                  fi

                  ebsToMountVolumeId=$(echo "$ebsToMount" | jq -r .VolumeId)

                  echo "Waiting for EBS to be available"
                  aws --region=${AWS::Region} ec2 wait volume-available --volume-ids "$ebsToMountVolumeId"

                  echo "Attaching EBS volume to this instance"
                  aws --region=${AWS::Region} ec2 attach-volume --volume-id "$ebsToMountVolumeId" --instance-id "$instanceId" --device "$deviceName"

                  echo "Waiting..."
                  aws --region=${AWS::Region} ec2 wait volume-in-use --volume-ids "$ebsToMountVolumeId"

                  echo "Checking OS for device availablity"
                  until [[ $(file -bsL $deviceName) != "cannot open"* ]]; do
                    sleep 2
                  done

                  echo "Checking EBS for existing filesystem"
                  if [[ ! $(blkid $deviceName) ]]; then
                    echo "No filesystem found; creating ext4 filesystem"
                    mkfs -t ext4 $deviceName
                  fi

                  echo "Creating mount point"
                  mkdir -p $mountPoint

                  echo "Adding to fstab"
                  echo "UUID=$(blkid -o value -s UUID $deviceName)  $mountPoint  ext4  defaults,nofail  0  2" >> /etc/fstab

                  echo "Mounting EBS"
                  mount -a

                  if [[ -n $existingEbsSnapshotId ]]; then
                    echo "Deleting old EBS snapshot and volume"
                    aws --region=${AWS::Region} ec2 delete-snapshot --snapshot-id "$existingEbsSnapshotId"
                    aws --region=${AWS::Region} ec2 delete-volume --volume-id "$existingEbsVolumeId"
                  fi

                  echo "EBS mount complete"