Jira server backups with a system imaging tool

[newaters]

Our IT Dept has installed a system snapshot / imaging tool ( AppAssure/RapidRecovery) on our Jira, Bitbucket, and Confluence servers. Are there any pitfalls to be aware of when relying on such a tool for system backups? Is it still preferable to perform database backups as discussed here, or using the bitbucket command line backup utility?

[Matt Doar · Accepted solution]

When you restore a snapshot, make sure it is not reading email from mailboxes used by production Jira. Also make sure it is accessing a snapshot of the DB, not live prod DB on another server

[newaters]

Thank you for your thought provoking response. For what its worth, the DB server is running on the same VM as the Jira Server. We are using PostgreSQL.

Seems like I would need to shutdown or otherwise lock the application to accomplish either of those two things. Does a mechanism to do that even exist?

Alternatively, if ever restoring from these backups I suppose I could destroy the database and restore it from a backup if I were separately doing daily backups of the database just before the system is imaged.

[Matt Doar]

I've seen it work with the DB on the same machine, though that is only recommended for smaller Jira instances. You'd need to check that the restored snapshot can still access its local DB and no other DB though.

Stopping email getting read could be done by disabling all mail habndlers before taking a snapshot, but that's not convenient. Changin the JVM parameters used at Jira startup would be better. So you don't want to have Jira restart automatically

[Vickey Palzor Lepcha]

You might also want to check the type of backup being taken - applications at times do not get restored on an incremental backup but would require a full backup.

[newaters]

I was told by the IT Dept. that the backups would be differential. I am not terribly familiar with this particular tool, but I can't conceive how you would do anything other than a full backup of system memory (if anything), regardless of whether you do a differential or incremental backup of the disc.

[Vickey Palzor Lepcha]

Taking full backups everyday would not be a good thing to do as it consumes a lot of space- rather once a week. One full backup + differential is good for restoring.

[newaters]

Agreed. I was speaking specifically about what is in system memory. I am not sure how you could possibly hope to deferentially back that up. A differential backup could very likely be close to a full backup from day to day.

[Vickey Palzor Lepcha]

Nick - by system memory what are you exactly hinting at ? Are you talking about filesystems - files and directories and the data, or something more discreet ?

[newaters]

This utility claims to take a snapshot of a system that is running, and when you restore that snap shot all of the applications that were previously running should pick up right where they were when the snapshot was taken.

So all volatile program memory in ram, the page file, etc, would be backed up too, including database caches. There are apparently some special integrations with this utility and MSSQL server that allow it to wait until database writes are done and perform a snapshot after caches are flushed. I haven't yet seen any documentation that suggests a similar integration with PostgreSQL.

Edit: I am not sure if anything I said above in the past two paragraphs is correct. I may have been misunderstanding their website/datasheet. As I said, I am not super familiar with this tool. Today is the first time I've ever heard about it.

[Vickey Palzor Lepcha]

ok Nick - sounds like it takes a complete server backup that would include JIRA and everything else that is installed.

I take a backup of just my jira or confluence schema in the backup - maybe that will take less time and would also let me be selective about what I am doing a backup of.

Your backup team would be more interested in taking a complete server backup - rather than just of the application.

[newaters]

Yes, our IT group isn't down in the weeds of knowing how to administrate each server. They provide comprehensive backups of all of our production servers. So I am basically wondering if that helps me much at all. I am wondering if I can trust the whether or not the database and Jira's other application data to not be corrupt or otherwise unusable because the backup was performed while Jira was running.

[Vickey Palzor Lepcha]

• You can probably consult your IT Team to take a selective backup of JIRA filesystem alone- I am sure there must be a provision - it would keep your backup separate from all other backups of the server. 
• You can ask your Database Team to take a schema backup of your JIRA schema only.

Both methods would keep your restoration process simple - you will not have to restore the complete server just to restore JIRA. 

And I am sure you have some kind of a restart procedure too in hand - that is when backups can happen. Normally users have a DB restarts in place nightly when backups are taken. You can push in your request too to have schema backups during that time. And DB guys normally shutdown DB for backups - this should correspond with your JIRA restart too.

[newaters]

Thanks Vickey, that is good advice.

[Ernie M.]

FWIW I posted along these lines. I have experience with this kind of endeavor in our internal private cloud on VMWare. Its not always so simple. The notion of having the application itself to be notified to be quiescent and flush to disk at the time of the snapshot is important. We have seen this not go well if the application is not directly VM aware or integrated (I.e. VSS, VMWare tools, Open VM tools, OVirt backup/restore API). We also investigated this with Red Hat recently and found them to agree at this. We spoke with Rubrik, Veeam, and Dell EMC and found pretty much all the same answers. VM based snapshot backups through CBT is attractive. It can simplify restores to a mouse click and minutes away. So these items are what has complicated our backup strategy for our Jira/Confluence install w/PostgreSQL, and why I created my post.