Is anybody doing zero-downtime on-prem Confluence backups on a large instance

[Steve]

Following all the guidance I could find on the web, I've implemented a backup strategy that takes our confluence instance down for 45 minutes to dump the database, then tar up the Atlassian filesystem. 

Does anybody know of a better way to take a zero downtime full backups of confluence?  It's really starting to impact our night-shift's ability to get to mission critical docs.

We're thinking of going so far as to restore the nightly backups to a completely different server in a different AWS region after the system comes back online, and have that be our failover for the following nights outage.

Thoughts? Opinions? Advice?

Thanks!

Steve

[Shankar Asam {Appfire}]

Steve,

Couple of ideas, you can even take the database dump while your confluence is up and running - use a script to do that on a nightly basis. If it's a linux, use tar to backup your file system too.

If it's a VM, try using snapshot based backups for DB and Application.

Another one, you can setup a database in master-slave mode (another db server on a different host and enable replication from master to slave) - this way the real time replication will happen for DB and use rsync to copy attachments or home directory as well.  you can enable backups on slave server where the replication / rsync is happening. You can also try exploring storage level backup options in case for SAN and NAS.

-Shankar

[Steve]

Shankar,

Thanks for the reply..

For all our systems we try to do three levels of DR:

1. AWS ASG automatic instance replacement of an unhealthy instance. All our data resides on a dedicated EBS volume, so ASG can simply detach the EBS from the unhealthy instance, build a new instance, and attach the ASG.
2. Daily AWS EBS Snapshots that we can use in case our primary EBS is corrupted somehow.
3. Fallback to a full, manual system restoration from backups (old school DR).

#3 is the one I'm currently trying to resolve. My primary concern is "how do I guarantee that the "DB Dump" and the tar of the Atlassian directory are in sync in the backup. My current solution is to set up a full outage of our instance on a nightly basis.

If there was a way to put the entire instance in "maintenance mode" (read-only), I'd feel comfortable with doing the DB dump and the tar with the system "up". The only things that could change in this state would be the log files, and I'm good with those being out of sync.

I'll keep trying to find a way to satisfy our "worst case restore scenario" and will post final results here when we get there.