Automate online site-backups for Jira and Confluence without programming

[#1]

Here I'll introduce a way to automate site-backups of Jira Cloud and Confluence Cloud with Jira Automation.

(0) Why site-backup is important

Atlassian doesn't officially provide data rollback based on customer requests. So it's essential to take site-backup regularly. For example, the backup will help you when any of site data was lost by the user's operation. Refer to the following part in Data storage FAQ for more details:

Atlassian cloud sites don't support the use of backup data to roll back changes.

Let me clarify what's covered by the approach introduced in this article. It's good as a first step while there are some constraints, such as:

• Prerequisite: You need to subscribe to at least one Jira family product; meaning it's unable to follow this instruction if you only with Confluence subscription as we'll utilize Jira Automation feature which is only available for Jira.
• Importantly, in this approach, we skip downloading the created site-backups to local machine simply because it's better than nothing. It is recommended for sure to download site-backups if you can. To do so, use the bash scripts listed below:
  • Backup script for Jira Cloud
  • Backup script for Confluence Cloud

(1) Preparation

First things first, you always need to have credentials when you do something important:

1. Issue an API token of site-admins account at https://id.atlassian.com/manage/api-tokens
2. Encode your credentials "$EMAIL_ADDRESS:$API_TOKEN" with base64 as (1-2) Example below. Or go with any online tool like Base64 Encode. Make sure not to include a line break at the end.
   • 

(1-2) Example - How to encode your API token

$ echo -n "site-admin@example.com:<YOUR_TOKEN_HERE>" | base64
c2l0ZS1hZG1pbkBleGFtcGxlLmNvbTo8WU9VUl9UT0tFTl9IRVJFPg==

(2) Create a rule for Confluence site-backup

Although you can start with either Jira or Confluence, we'll look into how to configure a rule for Confluence here for convenience.

1. Set up the schedule
   • Add the content of (2-1) below to Cron expression
   • Select "simply run the conditions and actions without providing issues" for When rule executes
   • 
2. Add an action - Send web requests
   • Webhook URL: {{baseUrl}}/wiki/rest/obm/1.0/runbackup
   • Headers:
     • Authorization: Basic <The output we got from (1-2)>
   • Webhook body: refer to (2-2)
   • 
3. Add another action - Log action
   1. To be able to keep tracking of what we got with the webhook request, specify {{webhookResponse}} in Log message
4. Name the rule "Site backup - Confluence Cloud" and save the setting
5. Do a test run with [Run rule] button
   • The log then will be available at Audit log
   • 
6. Check the download link
   • 

(2-1) Cron expression

Each backup should be performed after 48 hours interval at least. We here are going to schedule the execution twice a week on Monday and Thursday. Also make sure to avoid maintenance windows of your site. For example, suppose your company sits in EST, specify 9 am in UTC:

0 0 9 ? * 1,4

(2-2) Webhook body

{"cbAttachments":"true" }

(3) Create a rule for Jira site-backup

I'll put the diff of settings as it's almost same with Confluence.

1. Schedule - same
2. Action - Send web request
   • Webhook URL: {{baseUrl}}/rest/backup/1/export/runbackup
   • Webhook body: refer to (3-1)
3. Remaining procedures - same
   • 

(3-1) Cron expression

Same here, on Monday and Thursday at 9 am in UTC:

0 0 9 ? * 1,4

(3-2) Webhook body

{"cbAttachments":"true", "exportToCloud":"true"}

(4) Constraints

Other than noted above, there are several points you should be aware of:

• Jira has an upper bound on rule executions per subscription plan. That means you will not get a backup if exceeded the limit. So consider upgrading to Premium, especially if you already have a respectable number of existing global automation rules.
• The backup link will be available for 7 days. The link then will be replaced with the next one. That means you will only be able to retrieve the backup file until the next backup ends. So make sure to download the backup file as soon as you found any data loss; otherwise, the backup link will be replaced with the one taken after the data loss.

[Veera]

Great job, @Kenta Yamamoto!!! 

[Taranjeet Singh]

very useful information indeed. Thanks!

[ITS_Elvadata]

is it possible to create a backup and download the file automatically via a Windows Server ?

Thanks

[Kenta Yamamoto]

Yes, it is possible. Refer to atlassianlabs/automatic-cloud-backup and try the PowerScript or Python script.

[ITS_Elvadata]

thx, i will try it :-)

[Thomas Kringstad]

Thanks, looks like this works.. but I have one question...in the Log Action is the correct input {webhookResponse}} or {webhookResponse.body}} 

In the png you have written .body but not in the text...

[Florin Istrate]

Any idea how to change the limit of the backups triggered from 48 hours to let's say every hour?

[Nivruti Vora]

@Florin IstrateI don't think it's possible(depending on what you want to backup). Check https://confluence.atlassian.com/jirakb/how-to-automate-backups-for-jira-cloud-applications-779160659.html

From the article: "You can perform data-only backups—excluding attachments, avatars, and logos—one after the other, without a delay. When you're creating a backup that includes attachments, avatars, and logos, though, you need to wait 48 hours from the when you last created a backup." It's a limitation!

[Kenta Yamamoto]

@Thomas Kringstad - As the response is different between Jira and Confluence, I'd recommend you to use {{webhookResponse}} for both to avoid the confusion. It will include all the headers and body content.

[Kenta Yamamoto]

@Florin Istrate - As Nivruti kindly suggested, there's no option to change the backup frequency limit at the moment. Consider to vote for these feature requests:

• https://jira.atlassian.com/browse/CLOUD-6498
• https://jira.atlassian.com/browse/CLOUD-6617

If you are working on something that requires frequent backups such as data migration, reach out to https://support.atlassian.com/contact/ so that Atlassian Support can temporarily increase the quota.

[Florin Istrate]

@Kenta Yamamoto - I have an open case with support, thank you!

[Jeffrey O_Hara]

Configured the backups for JIRA Cloud instance using Automation Rules and it works great. 

However, I would like to download the backups locally since they're overwritten. 

Configured the Jira-backup.sh script from atlassianlabs/automatic-cloud-backup link above. However, I'm receiving the error "Basic auth with password is not allowed on this instance". I see this link to setup oath for JIRA Cloud https://developer.atlassian.com/cloud/jira/platform/jira-rest-api-oauth-authentication/?_ga=2.101848449.109137584.1587588476-1332438393.1568149174. How can I tell which authorization I'm using? I don't have SAML authentication configured. 

[Christopher Harris]

Even with all these steps, it still seems that one has to manually download the backup file.  What exactly is being accomplished with the above steps?  Simply generating a backup file, this is achievable with one click in backup manager.  What we really need is automatic backups, end to end.  But what we really really need, is Jira to build in the ability to recovery deleted data at least for some period of time.

[Kenta Yamamoto]

@Jeffrey O_Hara - Create an API token at https://id.atlassian.com/manage-profile/security/api-tokens and specify it as API_TOKEN.

[Kenta Yamamoto]

@Christopher Harris - I understand your concern and do agree with you. The workaround introduced in this article is far from the ideal experience. Please go with other solutions like below:

• Backup script for Jira Cloud
• Backup script for Confluence Cloud

what we really really need, is Jira to build in the ability to recovery deleted data at least for some period of time. 

As this feature is not yet implemented, I'd advise you to add yourself as a watcher to the feature request tickets below to receive any updates and add any comments you think are necessary and/or important to the feature request. I've also added a private comment linking this case to the feature request for our dev's review.

• https://jira.atlassian.com/browse/CLOUD-6498
• https://jira.atlassian.com/browse/CLOUD-6617

In addition, more information about how Atlassian implements New Features can be found in Implementation of New Features Policy.

[KP]

Awesome, @Kenta Yamamoto !

[Darryl Lee]

Hi, I've setup the Automation Rule for backups and it has run twice now (and the logs show that it was successful). But each time I've gone to the Backup manager page to look (usually a day or so after the rule runs), I do not see a download link.

I thought that backups are supposed to be kept for 7 days.

I just manually ran the rule, and the "Create backup for cloud" button is grayed out, so I assume a backup is in process, but I see no progress bar, and I suspect I will have the same issue again of no download link when the backup is complete.

Any ideas?

[Ooof, I realized I forgot to output the {{webhookResponse}} to the Audit log. Perhaps that will have a URL for download/progress that I'm missing. Still odd about the link not showing up in the Backup manager page. This is for Jira, BTW.]

[Darryl Lee]

Ah, I think the trick was that I forgot this in the Webhook body: 

"exportToCloud":"true"

INTERESTINGLY... the backup was created, and it is supposedly Server compatible, and I *DID NOT* have to delete my Next-Gen project (which isn't an option for us).

It was a pain to find the download though. I had to hit up this URL:

> https://{MYHOST}.atlassian.net/rest/backup/1/export/lastTaskId

Which provided the TaskId of the last backup I kicked off. So then I did:

> https://{MYHOST}.atlassian.net/rest/backup/1/export/getProgress?taskId={LASTTASKID}

Which showed this result

{
status: "Success",
description: "Cloud Export task",
message: "Completed export",
result: "export/download/?fileId=MYFILEID",
progress: 100,
exportType: "SERVER",
}

So then, the final download link is:

> https://${INSTANCE}/plugins/servlet/export/download/?fileId=MYFILEID

And... proof being in the pudding, I was able to successfully import this backup file to a Server instance. Woot!

UPDATE
After the import, I couldn't login, which is a known issue after importing from Cloud. So, per the instructions here, I restarted with the atlassian.recovery.password property set.

For some reason, restarting with this property set triggered Jira to do an immediate reindex.

Unfortunately this reindex failed with 1 error:

2020-09-24 14:58:46,125-0700 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.LicenseCheckingUpgradeService] An exception occurred when running to call a callable method, this shouldn't happen.

java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors

Finally though, after another stop and restart, Jira did successfully start up and I was able to login with the recovery_admin account/password. But meh, no data. Ok, will continue to dig into it.