JIRA/Confluence backup script - authentication error

[CeDeAdmin]

Hi everybody

I'm trying to get the PowerShell script running again, which doesnt work anymore for a few months. After replacing the URLs of the old Cloud infrastructure with the new ones mentioned here - https://jira.atlassian.com/browse/JRACLOUD-67169?src=confmacro&_ga=2.152189625.1813114333.1508752703-597319643.1507625905 - I get the next error, when requesting the backup, progress status and download: Invoke-RestMethod : Der Remoteserver hat einen Fehler zurückgegeben: (401) Nicht autorisiert.
At D:\Scripts\Jira Cloud Backup - new URLs.ps1:19 char:1
+ Invoke-RestMethod -Method Post -Uri "https://$hostname/rest/backup/1/ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Using these PS commands:

# Login
Invoke-WebRequest -Method Post -Uri "https://$hostname/login" -SessionVariable $session -Body @{username = $username; password = $password} | Out-Null

# Request backup
Invoke-RestMethod -Method Post -Uri "https://$hostname/rest/backup/1/export/runbackup" -WebSession $session -ContentType 'application/json' -Body (@{cbAttachments = $attachments} | ConvertTo-Json -Compress) | Out-Null

Can anyone help me out?

[Dario B]

Hi @CeDeAdmin,

The described behaviour is the same as:

• https://jira.atlassian.com/browse/JRACLOUD-46830
• https://jira.atlassian.com/browse/ID-6351

The solution/workaround for both bugs is the same:

Most of the times this happens because of the bruteforce protection mechanism. A way to have this working again is to reset the password for the user in id.atlassian.com by going to:
- https://id.atlassian.com/manage/change-password

If this is not applicable because you didn't have a password previously set in id.atlassian.com and therefore you cannot use the change-password form. Then do the following:

1. Go to https://id.atlassian.com/login/resetpassword
2. Enter you email, complete the CAPTCHA and click on Send
3. Once you receive the password reset email, do not click on the button to Access but on the link below that says "reset password".
4. Fill up the new password and you are done.

Please try the above steps and let me know if this works. If it doesn't either provide more details or open a support request in support.atlassian.com to have this issue addressed.

Cheers,
Dario

[CeDeAdmin]

Hi Dario

I changed the password for the user user in the backup script, but I get the same error back in the event log.
I couldnt find the error in the PowerShell script. I posted the two commands before, which log in in a first step and request to start the backup. The login session should be saved in the first step in the variable $session and is used in the backup-request-command. Maybe you see there an error, if you know PowerShell?

I also opened a ticket for this case - JST-336037. But Travis, which is in charge there, sent me URL to the page with the curl commands and new URL of the new infrastructure, but this is not helping me out with the authentication problem. I'm also waiting there for a reply.

Regards, Stefan

[CeDeAdmin]

And I also tried out the curl commands over GIT Bash, and they seem to work, but I get no result back. Used the commands mentioned in JRACLOUD-67169 (took only the first line of command, second line seems to be the result of the command - Sorry, I'm pretty new to scripting so I rather ask than stay unsure) Maybe it doesnt give me the backup file as result, as I made a manual Cloud Backup today and downloaded it, to have something until the automatic backup works again...

[Dario B]

@CeDeAdmin 

I checked the support ticket and there is no information in there mentioning the fact that you have issues with the authentication and not with the endpoints or with the script itself. Also, you haven't mentioned this community thread in the support ticket. That's why you have been sent back to here by the assigned engineer.

I left an  internal  comment saying that your issue is actually with authentication. However, by re-reading your error message it seems that the unauthorised error is coming from the line that is starting the backup and not from the one to start a session and save it in a cookie-jar file.. at this point I am wondering if the problem is not with the way the cookie is stored/used by the script itself... 

Have you actually tried to run the rest calls in a shell one by one using curl (and not in a script) and see if it works that way? 

This is because, as written in the section named Support Does Not Include of the Atlassian Support Offerings :

Support Does Not Include

Development questions or requests.

• For community-based development support, please post these questions on Atlassian Community.
• Note: If a bug in Atlassian development API is believed to be the root cause, sample code that demonstrates the problem and can be replicated must be provided.

Please try to run the commands from the workaround for JRACLOUD-67169 one after the other from command line to identify where the problem is. Also, provide this information in the support ticket.

Cheers,
Dario

[Dario B]

Sorry I missed you second reply.. Please provide the output of the commands you have run to the support engineer.

Also, provide more details on what you mean with:

and they seem to work, but I get no result back

If it works, then you should get the same results as the examples in the bug ticket.. Some error message otherwise. 

Provide also the email address of the account used to authenticate the REST calls to @Travis D.  so that he can try to replicate the authentication issue.

[CeDeAdmin]

Ye I first opened the ticket to ask for help, after I tried out some of the community suggested solutions. When I realized I wont get support to that scripting problem, I opened this community thread. And I mentioned the authentication error in the ticket comment of the 30/Oct/17 9:47 AM, which you might overlooked.

However, in PowerShell I can run the login command successfully, but the second command for starting the backup..

---START---

$account     = 'xx'
$username    = 'xx'
$password    = 'xx'
$destination = 'xx'
$attachments = $true
$hostname    = "$account.atlassian.net"
$today       = Get-Date -format yyyyMMdd
New-Object System.Management.Automation.PSCredential($username, (ConvertTo-SecureString $password -AsPlainText -Force))

# Login
Invoke-WebRequest -Method Post -Uri "https://$hostname/login" -SessionVariable session -Body @{username = $username; password = $password} #| Out-Null

# Request backup
$result = Invoke-RestMethod -Method Post -Uri "https://$hostname/rest/backup/1/export/runbackup"  -WebSession $session -ContentType 'application/json' -Body (@{cbAttachments = $attachments} | ConvertTo-Json -Compress) | Out-Null

---END---

..fails with the Error "401Client must be authenticated to access this resource." The URI is correct, I checked over the ticket on which cloud infrastructure we are - new one.

---------------------------------------------------------------

With curl it looks like a successful login and start-backup command (I logged in with the Atlassian account and generated a cookies.txt, containing the login date. So I guess I dont need the login (first)command anymore, right?), but it fails to check the progress of the Backup with the authentication error:

--START--

xx MINGW64 ~
$ curl -Body D:\cookies.txt -Headers "Content-Type: application/json" -d '{"username":"xx", "password":"xx" }' -X POST https://xx.atlassian.net/rest/auth/1/session
curl: (3) Port number ended with 'c'
curl: (3) Port number ended with ' '
% Total    % Received % Xferd  Average Speed   Time    Time Time  Current Dload  Upload   Total   Spent    Left  Speed
100    68    0     0  100    68      0     68  0:00:01 --:--:--  0:00:01    89

xx MINGW64 ~
$ curl -b D:\cookies.txt -H "Accept: application/json" -Headers "Content-Type: application/json" https://xx.atlassian.net/rest/backup/1/export/runbackup --data-binary '{"cbAttachments":"true", "exportToCloud":"true"}'
curl: (3) Port number ended with ' '
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
100    48    0     0  100    48      0     48  0:00:01 --:--:--  0:00:01   105

xx MINGW64 ~
$ curl -b D:\cookies.txt -H "Accept: application/json" -Headers "Content-Type: application/json" https://xx.atlassian.net/rest/internal/2/task/progress/10075
curl: (3) Port number ended with ' '
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
100    85    0    85    0     0     85      0 --:--:-- --:--:-- --:--:--   151{"message":"Client must be authenticated to access this resource.","status-code":401}

---END--

So I dont understand why the cookies.txt seems to work for the backup request itself but not for checking the progress anymore..

[CeDeAdmin]

I just checked the backup-manager of the URL I requested a backup and downloaded it to check the creation date - the backup request was successful. So it seems that the login + backup request command of curl worked.
What is left failing, is the progress check (I guess I have to make a loop for the backup progress request with a while/if, so when the state is 100% it starts the download command?) and also the .zip file download command - each created Backup ZIP file will have a different download link (correct me please if I'm wrong) or does it stay static? How can I provide that the curl command get the right link for each new new generated backup file if the download URL changes?

Thanks & kind regards, Stefan

[CeDeAdmin]

Sorry for making again a split reply, but I already tested the curl download command in advance and it partially works. A file is downloaded, but it is corrupt (It is always 65KB, manually downloaded it has 426KB). I copied the link of the created backup ZIP file and pasted it into the curl command shown below.

$ curl -Body D:\cookies.txt https://xx.atlassian.net/plugins/servlet/export/download/21a90dbc-21cc-457f-a47e-d93ebf85b51b/export-manifest5476147036890758686.json -o D:\_JIRA_BACKUP_CEDESHOPTEST.zip

[Dario B]

try to add the -L option to curl so that it follows redirect.

Also, I have just noticed that you are passing to curl the -Body parameter that does not exist, instead of -b (or --cookie, see the curl man page below):

• https://curl.haxx.se/docs/manpage.html

For the rest, please follow-up in the support ticket.

[Chris Dunne _Raledo_]

Hi Stefan, did you find a solution to this? I am having the same problem.

Thanks

Chris

[Dario B]

@Chris Dunne _Raledo_ 

 Have you tried adding the -L option to the curl command?

Or you are referring to the authentication error (it is not clear what is the same problem you are having)? 

Cheers,

Dario

[Chris Dunne _Raledo_]

Thanks for the quick reply - I am referring to the authentication error. 

[Dario B]

Have you set your password in id.atlassian.com as already explained? 

Also, have you tried authenticating a simple REST call using curl and the same access credentials?

E.g. Please run below command and paste it together with the output in here (remove your email address and password).

curl -D- -u EMAIL:PASSWORD -H "Accept: application/json" -H "Content-Type: application/json" -X GET  https://INSTANCE/rest/api/2/mypermissions

[CeDeAdmin]

Hi Chris,
it is not working yet. I hope I can get the curl commands running with the help here. As workaround there is an UNIX based script, that seems to work better. But I dont know how to make it run with Windows and creating a virtual "backup" Linux machine only for this task seems a bit like an overkill, so I'm looking for some compatibility command/script application.

Hi Dario,
- Password is set on id.atlassian.com and I even reset it a few weeks ago to make sure it's not this kind of password problem
- Does a REST call look like this?:
$ curl -b D:\cookiesjira.txt -H "Accept: application/json" - Headers "Content-Type: application/json" https://INSTANCE/rest/backup/1/export/runbackup --data-binary '{"cbAttachments":"true", "exportToCloud":"true"}'
curl: (3) Port number ended with ' '
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    48    0     0  100    48      0     48  0:00:01 --:--:--  0:00:01    58
But when I change the cookie parameter name to a wrong one I get the same result, therefore it looks like it's not really using it. I copied this command out of another thread here and replaced the instance name and created the cookie file with the "cookies.txt" addon in Chrome, after logging in and navigating to the backup manager page of Confluence/Jira (theoretically the same cookie should work for Jira & Confluence login?).

And I added the -L parameter, but the output is the same. Or is there a specific position of this parameter needed? Also changed back the -Body parameter to "-b". That happened when I used the commands with PowerShell and tried to complete these short-parameters, so PowerShell understands it.

[CeDeAdmin]

- run your last command over GIT Bash:

$ curl -D- -u "EMAIL":"PASSWORD" -Headers "Accept: application/json"                                  -Headers "Content-Type: application/json" -X GET  https://INSTANCE/rest/api/2/my                                 permissions
curl: (3) Port number ended with ' '

curl: (3) Port number ended with ' '

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0HTTP/2 401

server: Atlassian Proxy/0.1.140

www-authenticate: OAuth realm="https%3A%2F%2FINSTACE"

atl-vtm-backend-time: 1248

content-type: text/html;charset=UTF-8

strict-transport-security: max-age=315360000; includeSubDomains; preload

date: Mon, 20 Nov 2017 07:20:47 GMT

x-arequestid: 51c7c8bd-0c88-4b8a-8b31-eb79aa8302b5

atl-vtm-time: 1248

x-content-type-options: nosniff

x-seraph-loginreason: AUTHENTICATED_FAILED

atl-vtm-queue-time: 0

I CUT THE HTML PART, AS IT DOES NOT LET ME POST IT IN HERE, NOT EVEN AS CODE FORMAT.

 Used the URI directly in the web browser, being logged in manually:

{"permissions":{"VIEW_WORKFLOW_READONLY":{"id":"45","key":"VIEW_WORKFLOW_READONLY","name":"View Read-Only Workflow","type":"PROJECT","description":"admin.permissions.descriptions.VIEW_WORKFLOW_READONLY","havePermission":false,"deprecatedKey":true},"CREATE_ISSUES":{"id":"11","key":"CREATE_ISSUES","name":"Create Issues","type":"PROJECT","description":"Ability to create issues.","havePermission":true},"VIEW_DEV_TOOLS":{"id":"29","key":"VIEW_DEV_TOOLS","name":"View Development Tools","type":"PROJECT","description":"Allows users in a software project to view development-related information on the issue, such as commits, reviews and build information.","havePermission":false},"BULK_CHANGE":{"id":"33","key":"BULK_CHANGE","name":"Bulk Change","type":"GLOBAL","description":"Ability to modify a collection of issues at once. For example, resolve multiple issues in one step.","havePermission":true},"CREATE_ATTACHMENT":{"id":"19","key":"CREATE_ATTACHMENT","name":"Create Attachments","type":"PROJECT","description":"Users with this permission may create attachments.","havePermission":true,"deprecatedKey":true},"DELETE_OWN_COMMENTS":{"id":"37","key":"DELETE_OWN_COMMENTS","name":"Delete Own Comments","type":"PROJECT","description":"Ability to delete own comments made on issues.","havePermission":true},"WORK_ON_ISSUES":{"id":"20","key":"WORK_ON_ISSUES","name":"Work On Issues","type":"PROJECT","description":"Ability to log work done against an issue. Only useful if Time Tracking is turned on.","havePermission":false},"PROJECT_ADMIN":{"id":"23","key":"PROJECT_ADMIN","name":"Administer Projects","type":"PROJECT","description":"Ability to administer a project in Jira.","havePermission":true,"deprecatedKey":true},"COMMENT_EDIT_ALL":{"id":"34","key":"COMMENT_EDIT_ALL","name":"Edit All Comments","type":"PROJECT","description":"Ability to edit all comments made on issues.","havePermission":true,"deprecatedKey":true},"ATTACHMENT_DELETE_OWN":{"id":"39","key":"ATTACHMENT_DELETE_OWN","name":"Delete Own Attachments","type":"PROJECT","description":"Users with this permission may delete own attachments.","havePermission":true,"deprecatedKey":true},"WORKLOG_DELETE_OWN":{"id":"42","key":"WORKLOG_DELETE_OWN","name":"Delete Own Worklogs","type":"PROJECT","description":"Ability to delete own worklogs made on issues.","havePermission":false,"deprecatedKey":true},"CLOSE_ISSUE":{"id":"18","key":"CLOSE_ISSUE","name":"Close Issues","type":"PROJECT","description":"Ability to close issues. Often useful where your developers resolve issues, and a QA department closes them.","havePermission":true,"deprecatedKey":true},"MANAGE_WATCHER_LIST":{"id":"32","key":"MANAGE_WATCHER_LIST","name":"Manage Watchers","type":"PROJECT","description":"Ability to manage the watchers of an issue.","havePermission":true,"deprecatedKey":true},"VIEW_VOTERS_AND_WATCHERS":{"id":"31","key":"VIEW_VOTERS_AND_WATCHERS","name":"View Voters and Watchers","type":"PROJECT","description":"Ability to view the voters and watchers of an issue.","havePermission":true},"ADD_COMMENTS":{"id":"15","key":"ADD_COMMENTS","name":"Add Comments","type":"PROJECT","description":"Ability to comment on issues.","havePermission":true},"COMMENT_DELETE_ALL":{"id":"36","key":"COMMENT_DELETE_ALL","name":"Delete All Comments","type":"PROJECT","description":"Ability to delete all comments made on issues.","havePermission":true,"deprecatedKey":true},"CREATE_ISSUE":{"id":"11","key":"CREATE_ISSUE","name":"Create Issues","type":"PROJECT","description":"Ability to create issues.","havePermission":true,"deprecatedKey":true},"DELETE_OWN_ATTACHMENTS":{"id":"39","key":"DELETE_OWN_ATTACHMENTS","name":"Delete Own Attachments","type":"PROJECT","description":"Users with this permission may delete own attachments.","havePermission":true},"DELETE_ALL_ATTACHMENTS":{"id":"38","key":"DELETE_ALL_ATTACHMENTS","name":"Delete All Attachments","type":"PROJECT","description":"Users with this permission may delete all attachments.","havePermission":false},"ASSIGN_ISSUE":{"id":"13","key":"ASSIGN_ISSUE","name":"Assign Issues","type":"PROJECT","description":"Ability to assign issues to other people.","havePermission":true,"deprecatedKey":true},"LINK_ISSUE":{"id":"21","key":"LINK_ISSUE","name":"Link Issues","type":"PROJECT","description":"Ability to link issues together and create linked issues. Only useful if issue linking is turned on.","havePermission":true,"deprecatedKey":true},"EDIT_OWN_WORKLOGS":{"id":"40","key":"EDIT_OWN_WORKLOGS","name":"Edit Own Worklogs","type":"PROJECT","description":"Ability to edit own worklogs made on issues.","havePermission":false},"CREATE_ATTACHMENTS":{"id":"19","key":"CREATE_ATTACHMENTS","name":"Create Attachments","type":"PROJECT","description":"Users with this permission may create attachments.","havePermission":true},"EDIT_ALL_WORKLOGS":{"id":"41","key":"EDIT_ALL_WORKLOGS","name":"Edit All Worklogs","type":"PROJECT","description":"Ability to edit all worklogs made on issues.","havePermission":false},"SCHEDULE_ISSUE":{"id":"28","key":"SCHEDULE_ISSUE","name":"Schedule Issues","type":"PROJECT","description":"Ability to view or edit an issue's due date.","havePermission":true,"deprecatedKey":true},"CLOSE_ISSUES":{"id":"18","key":"CLOSE_ISSUES","name":"Close Issues","type":"PROJECT","description":"Ability to close issues. Often useful where your developers resolve issues, and a QA department closes them.","havePermission":true},"SET_ISSUE_SECURITY":{"id":"26","key":"SET_ISSUE_SECURITY","name":"Set Issue Security","type":"PROJECT","description":"Ability to set the level of security on an issue so that only people in that security level can see the issue.","havePermission":true},"SCHEDULE_ISSUES":{"id":"28","key":"SCHEDULE_ISSUES","name":"Schedule Issues","type":"PROJECT","description":"Ability to view or edit an issue's due date.","havePermission":true},"WORKLOG_DELETE_ALL":{"id":"43","key":"WORKLOG_DELETE_ALL","name":"Delete All Worklogs","type":"PROJECT","description":"Ability to delete all worklogs made on issues.","havePermission":false,"deprecatedKey":true},"COMMENT_DELETE_OWN":{"id":"37","key":"COMMENT_DELETE_OWN","name":"Delete Own Comments","type":"PROJECT","description":"Ability to delete own comments made on issues.","havePermission":true,"deprecatedKey":true},"ADMINISTER_PROJECTS":{"id":"23","key":"ADMINISTER_PROJECTS","name":"Administer Projects","type":"PROJECT","description":"Ability to administer a project in Jira.","havePermission":true},"DELETE_ALL_COMMENTS":{"id":"36","key":"DELETE_ALL_COMMENTS","name":"Delete All Comments","type":"PROJECT","description":"Ability to delete all comments made on issues.","havePermission":true},"RESOLVE_ISSUES":{"id":"14","key":"RESOLVE_ISSUES","name":"Resolve Issues","type":"PROJECT","description":"Ability to resolve and reopen issues. This includes the ability to set a fix version.","havePermission":true},"VIEW_READONLY_WORKFLOW":{"id":"45","key":"VIEW_READONLY_WORKFLOW","name":"View Read-Only Workflow","type":"PROJECT","description":"Users with this permission may view a read-only version of a workflow.","havePermission":false},"ADMINISTER":{"id":"0","key":"ADMINISTER","name":"Jira Administrators","type":"GLOBAL","description":"Ability to perform most administration functions (excluding Import & Export, SMTP Configuration, etc.).","havePermission":true},"MOVE_ISSUES":{"id":"25","key":"MOVE_ISSUES","name":"Move Issues","type":"PROJECT","description":"Ability to move issues between projects or between workflows of the same project (if applicable). Note the user can only move issues to a project he or she has the create permission for.","havePermission":true},"TRANSITION_ISSUES":{"id":"46","key":"TRANSITION_ISSUES","name":"Transition Issues","type":"PROJECT","description":"Ability to transition issues.","havePermission":true},"SYSTEM_ADMIN":{"id":"44","key":"SYSTEM_ADMIN","name":"Jira System Administrators","type":"GLOBAL","description":"Ability to perform all administration functions. There must be at least one group with this permission.","havePermission":false},"DELETE_OWN_WORKLOGS":{"id":"42","key":"DELETE_OWN_WORKLOGS","name":"Delete Own Worklogs","type":"PROJECT","description":"Ability to delete own worklogs made on issues.","havePermission":false},"BROWSE":{"id":"10","key":"BROWSE","name":"Browse Projects","type":"PROJECT","description":"Ability to browse projects and the issues within them.","havePermission":true,"deprecatedKey":true},"EDIT_ISSUE":{"id":"12","key":"EDIT_ISSUE","name":"Edit Issues","type":"PROJECT","description":"Ability to edit issues.","havePermission":true,"deprecatedKey":true},"MODIFY_REPORTER":{"id":"30","key":"MODIFY_REPORTER","name":"Modify Reporter","type":"PROJECT","description":"Ability to modify the reporter when creating or editing an issue.","havePermission":true},"EDIT_ISSUES":{"id":"12","key":"EDIT_ISSUES","name":"Edit Issues","type":"PROJECT","description":"Ability to edit issues.","havePermission":true},"MANAGE_WATCHERS":{"id":"32","key":"MANAGE_WATCHERS","name":"Manage Watchers","type":"PROJECT","description":"Ability to manage the watchers of an issue.","havePermission":true},"EDIT_OWN_COMMENTS":{"id":"35","key":"EDIT_OWN_COMMENTS","name":"Edit Own Comments","type":"PROJECT","description":"Ability to edit own comments made on issues.","havePermission":true},"ASSIGN_ISSUES":{"id":"13","key":"ASSIGN_ISSUES","name":"Assign Issues","type":"PROJECT","description":"Ability to assign issues to other people.","havePermission":true},"BROWSE_PROJECTS":{"id":"10","key":"BROWSE_PROJECTS","name":"Browse Projects","type":"PROJECT","description":"Ability to browse projects and the issues within them.","havePermission":true},"VIEW_VERSION_CONTROL":{"id":"29","key":"VIEW_VERSION_CONTROL","name":"View Development Tools","type":"PROJECT","description":"Allows users to view development-related information on the view issue screen, like commits, reviews and build information.","havePermission":false,"deprecatedKey":true},"WORK_ISSUE":{"id":"20","key":"WORK_ISSUE","name":"Work On Issues","type":"PROJECT","description":"Ability to log work done against an issue. Only useful if Time Tracking is turned on.","havePermission":false,"deprecatedKey":true},"COMMENT_ISSUE":{"id":"15","key":"COMMENT_ISSUE","name":"Add Comments","type":"PROJECT","description":"Ability to comment on issues.","havePermission":true,"deprecatedKey":true},"WORKLOG_EDIT_ALL":{"id":"41","key":"WORKLOG_EDIT_ALL","name":"Edit All Worklogs","type":"PROJECT","description":"Ability to edit all worklogs made on issues.","havePermission":false,"deprecatedKey":true},"EDIT_ALL_COMMENTS":{"id":"34","key":"EDIT_ALL_COMMENTS","name":"Edit All Comments","type":"PROJECT","description":"Ability to edit all comments made on issues.","havePermission":true},"DELETE_ISSUE":{"id":"16","key":"DELETE_ISSUE","name":"Delete Issues","type":"PROJECT","description":"Ability to delete issues.","havePermission":true,"deprecatedKey":true},"MANAGE_SPRINTS_PERMISSION":{"id":"-1","key":"MANAGE_SPRINTS_PERMISSION","name":"Manage sprints","type":"PROJECT","description":"Ability to manage sprints.","havePermission":false},"USER_PICKER":{"id":"27","key":"USER_PICKER","name":"Browse Users","type":"GLOBAL","description":"Ability to select a user or group from a popup window as well as the ability to use the 'share' issues feature. Users with this permission will also be able to see names of all users and groups in the system.","havePermission":true},"CREATE_SHARED_OBJECTS":{"id":"22","key":"CREATE_SHARED_OBJECTS","name":"Create Shared Objects","type":"GLOBAL","description":"Ability to share dashboards and filters with other users, groups and roles.","havePermission":true},"ATTACHMENT_DELETE_ALL":{"id":"38","key":"ATTACHMENT_DELETE_ALL","name":"Delete All Attachments","type":"PROJECT","description":"Users with this permission may delete all attachments.","havePermission":false,"deprecatedKey":true},"DELETE_ISSUES":{"id":"16","key":"DELETE_ISSUES","name":"Delete Issues","type":"PROJECT","description":"Ability to delete issues.","havePermission":true},"MANAGE_GROUP_FILTER_SUBSCRIPTIONS":{"id":"24","key":"MANAGE_GROUP_FILTER_SUBSCRIPTIONS","name":"Manage Group Filter Subscriptions","type":"GLOBAL","description":"Ability to manage (create and delete) group filter subscriptions.","havePermission":true},"RESOLVE_ISSUE":{"id":"14","key":"RESOLVE_ISSUE","name":"Resolve Issues","type":"PROJECT","description":"Ability to resolve and reopen issues. This includes the ability to set a fix version.","havePermission":true,"deprecatedKey":true},"SERVICEDESK_AGENT":{"id":"-1","key":"SERVICEDESK_AGENT","name":"Service Desk Agent","type":"PROJECT","description":"Allows users to interact with customers and access Service Desk features of a project.","havePermission":false},"ASSIGNABLE_USER":{"id":"17","key":"ASSIGNABLE_USER","name":"Assignable User","type":"PROJECT","description":"Users with this permission may be assigned to issues.","havePermission":true},"TRANSITION_ISSUE":{"id":"46","key":"TRANSITION_ISSUE","name":"Transition Issues","type":"PROJECT","description":"Ability to transition issues.","havePermission":true,"deprecatedKey":true},"COMMENT_EDIT_OWN":{"id":"35","key":"COMMENT_EDIT_OWN","name":"Edit Own Comments","type":"PROJECT","description":"Ability to edit own comments made on issues.","havePermission":true,"deprecatedKey":true},"MOVE_ISSUE":{"id":"25","key":"MOVE_ISSUE","name":"Move Issues","type":"PROJECT","description":"Ability to move issues between projects or between workflows of the same project (if applicable). Note the user can only move issues to a project he or she has the create permission for.","havePermission":true,"deprecatedKey":true},"WORKLOG_EDIT_OWN":{"id":"40","key":"WORKLOG_EDIT_OWN","name":"Edit Own Worklogs","type":"PROJECT","description":"Ability to edit own worklogs made on issues.","havePermission":false,"deprecatedKey":true},"DELETE_ALL_WORKLOGS":{"id":"43","key":"DELETE_ALL_WORKLOGS","name":"Delete All Worklogs","type":"PROJECT","description":"Ability to delete all worklogs made on issues.","havePermission":false},"LINK_ISSUES":{"id":"21","key":"LINK_ISSUES","name":"Link Issues","type":"PROJECT","description":"Ability to link issues together and create linked issues. Only useful if issue linking is turned on.","havePermission":true}}}

Thanks for the help & regards, Stefan

[Dario B]

@CeDeAdmin,

In the same github repository that was pointed to you in the support request there is a power shell script as well:

- https://github.com/mattock/automatic-cloud-backup/blob/master/backup.ps1

Did you try it since you mentioned that the Bash script from the same repository works?

[Chris Dunne _Raledo_]

I am making some progress with this script. But its quite difficult to test this since you can only run the backup every 48 hours. Is there some other way we can test this API?

[CeDeAdmin]

@Dario BI tried this one and it doesnt work. Or better said partially.. it works until line 19 - where it actually demands for the backup. I outcommented the Out-Null parameter at line 16 to see, if the login itself works and yes, it does. But when running the backup command it says that I have to be authenticated..:
Invoke-RestMethod : 401Client must be authenticated to access this resource.

At D:\Backup\Atlassian\backup.ps1:19 char:1
+ Invoke-RestMethod -Method Post -Uri "https://$hostname/rest/backup/1/ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

So I dont understand what is going wrong here. Is the $session variable not taken into the next command? If yes, why not??

@Chris Dunne _Raledo_Hi Chris, did that also happen to you or does it work fine, without 401 error? If so, did you change something in the script except the login variables in the beginning?

[Chris Dunne _Raledo_]

Yes, I got the same results when running the powershell script. It didn't work for me for the very same reason. I have had some success with writing a Python version of the script. It seems to be working, but its difficult to test since you can only run your backup every 48 hours. 

[CeDeAdmin]

Such a pity. However, if you have success with your Python script, please let me know (and if I kindly ask, can I have it too?) :)

I'm trying to get the UNIX based script running with Windows or find an application, which translates the UNIX commands for Windows to understand.

[Chris Dunne _Raledo_]

Happy to share the script. Are you familiar with Python? Could you send me your email address?

[Dario B]

@CeDeAdmin @Christoph Hacker ,

First of all let's confirm again that you are both connecting yo a JIRA Cloud instance and not a Server one since it may be working differently in there.

Then, if you are using Cloud, then the script is failing at line 19 because it is calling the old endpoint: 

• https://$hostname/rest/obm/1.0/runbackup

 Instead of:

• https://$hostname/rest/backup/1/export/runbackup

Again, please compare the endpoints called in the script and the data sent with the ones in the workaround for https://jira.atlassian.com/browse/JRACLOUD-67169 since you already have everything you need in there.

[Chris Dunne _Raledo_]

I can confirm I am connecting to cloud.

Also, I can confirm that the powershell script is using the new endpoint. This is the version downloaded from the bitbucket repo - relevant section is 

# LoginInvoke-WebRequest -Method Post -Uri "https://$hostname/login" -SessionVariable session -Body @{username = $username; password = $password} | Out-Null
# Request backup#Invoke-RestMethod -Method Post -Uri "https://$hostname/rest/obm/1.0/runbackup" -WebSession $session -ContentType 'application/json' -Body (@{cbAttachments = $attachments} | ConvertTo-Json -Compress) | Out-Null$body = @{ cbAttachments='true' exportToCloud='true' }$bodyjson = $body | ConvertTo-JsonInvoke-RestMethod -Method Post -Headers @{"Accept"="application/json"} -Uri "https://$hostname/rest/backup/1/export/runbackup" -WebSession $session -ContentType 'application/json' -Body $bodyjson -Verbose | ConvertTo-Json -Compress | Out-Null$GetBackupID = Invoke-WebRequest -Method Get -WebSession $session https://$hostname/rest/backup/1/export/lastTaskId$LatestBackupID = $GetBackupID.content

[Dario B]

@Christoph Hacker

The endpoint you have copy-pasted is the old one that does not work anymore:

-Method Post -Uri "https://$hostname/rest/obm/1.0/runbackup"

Again, the new endpoint is:

• https://$hostname/rest/backup/1/export/runbackup

Also, the script is not calling the session endpoint as explained in https://jira.atlassian.com/browse/JRACLOUD-67169 but:

https://$hostname/login

However, if this works I have not too much to say about.

Again, please review the Workaround and the Comment sections of https://jira.atlassian.com/browse/JRACLOUD-67169 in order to have this working.

[Chris Dunne _Raledo_]

Dario

I think the code got mangled in pasting. the line with the old URL is commented out and the ps1 script is using the new URL.

Invoke-RestMethod -Method Post -Headers @{"Accept"="application/json"} -Uri "https://$hostname/rest/backup/1/export/runbackup" -WebSession $session -ContentType 'application/json' -Body $bodyjson -Verbose | ConvertTo-Json -Compress | Out-Null

This is directly from the Bitbucket repo.

I've abandoned these scripts and written my own in Python.

[Chris Dunne _Raledo_]

Dario

Could you tell me if Confluence Cloud has a similar set of endpoints and what they are?

Thanks

Chris

[Dario B]

Hi Chris, 

The endpoints are listed in the below comment to JRACLOUD-67169:

- https://jira.atlassian.com/browse/JRACLOUD-67169?focusedCommentId=1633803&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1633803

Again, please review the comments on JRACLOUD-67169 since most of the problems you are having, have already been discussed and resolved in there.

I hope this helps.

[CeDeAdmin]

@Dario B

- Yes we're using Cloud, not server

- I called the new endpoint since I checked in the ticket I had with Travis on which infrastructure we are hosted. I just tried it with the old URL /rest/obm/1.0/runbackup and it gives a different error: Dead link!
If I paste https://$hostname/rest/backup/1/export/runbackup into the web browser, I get a blank site. Since the error message is "401Client must be authenticated to access this resource." (it should use the session variable $session, but it doesnt! I made an output of the $session variable into a .txt file to see what it is containing  and it's empty. Therefore I get this error message, that I've to be authenticated.)

I'm trying to make this session variable work when I find time. Same authentication error exists with the curl Script, using a cookie file..
I think we are running in a circle Dario, the PS Script given simply doesn't work or the calls by it are denied of your company restrictions, I cant tell..

@Chris Dunne _Raledo_I dont know Python, but a co-worker understands the basics. Can you post it here? Else wise I'd be glad if you send it to stefan.engeli@cede.ch, if you don't want to post it here. I didn't find how to send a personal message to you.

Regards, Stefan

[CeDeAdmin]

@Chris Dunne _Raledo_, can you post the python script you made or send it to me?Sorry I don't find your post or comment for it, I worked a while on other things..

[Matt Roblin]

Hi - not sure if you were still looking for a PowerShell script - this ticket has been open in Bitbucket for a while and has some amended versions of the original script including one from this morning to handle the latest amendment to the REST API that Dario mentioned above ..

https://bitbucket.org/atlassianlabs/automatic-cloud-backup/issues/15/rest-url-is-coming-back-as-a-dead-link

[Dario B]

And this is the script for JIRA since some endpoints changed again.

All I did in order to have the backup script working again is:

1. Open the developer tools
2. Run the backup from the UI
3. Check the endpoints used and modify the existing script accordingly

#!/bin/bash

USERNAME=
PASSWORD=
INSTANCE=
LOCATION="/path/to/download/folder"

### Checks for progress 3000 times every 20 seconds ###
PROGRESS_CHECKS=3000
SLEEP_SECONDS=20

# Set this to your Atlassian instance's timezone.
# See this for a list of possible values:
# https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE=Europe/Amsterdam
 
##----START-----###
echo "starting the script"

# Grabs cookies and generates the backup on the UI. 
TODAY=$(TZ=$TIMEZONE date +%Y%m%d)
COOKIE_FILE_LOCATION=jiracookie
curl --silent --cookie-jar $COOKIE_FILE_LOCATION -X POST "https://${INSTANCE}/rest/auth/1/session" -d "{\"username\": \"$USERNAME\", \"password\": \"$PASSWORD\"}" -H 'Content-Type: application/json' --output /dev/null

## The $BKPMSG variable will print the error message, you can use it if you're planning on sending an email
BKPMSG=$(curl -s --cookie $COOKIE_FILE_LOCATION -H "Accept: application/json" -H "Content-Type: application/json" https://${INSTANCE}/rest/backup/1/export/runbackup --data-binary '{"cbAttachments":"true", "exportToCloud":"true"}' )


##ADDED##
echo "message: $BKPMSG"

 
#Checks if the backup procedure has failed
if [ "$(echo "$BKPMSG" | grep -ic error)" -ne 0 ]; then
rm $COOKIE_FILE_LOCATION
echo "FAILED, IT RETURNED: $BKPMSG"
exit
fi

TASK_ID=$(curl -s --cookie $COOKIE_FILE_LOCATION -H "Accept: application/json" -H "Content-Type: application/json" https://${INSTANCE}/rest/backup/1/export/lastTaskId)

#Checks if the backup exists every 10 seconds, 2000 times. If you have a bigger instance with a larger backup file you'll probably want to increase that.
for (( c=1; c<=2000; c++ ))
do
PROGRESS_JSON=$(curl -s --cookie $COOKIE_FILE_LOCATION https://${INSTANCE}/rest/backup/1/export/getProgress?taskId=${TASK_ID})
FILE_NAME=$(echo "$PROGRESS_JSON" | sed -n 's/.*"result"[ ]*:[ ]*"\([^"]*\).*/\1/p')

##ADDED##
echo "$PROGRESS_JSON"

if [[ $PROGRESS_JSON == *"error"* ]]; then
break
fi

if [ ! -z "$FILE_NAME" ]; then
break
fi
sleep 10
done

#If after 2000 attempts it still fails it ends the script.
if [ -z "$FILE_NAME" ];
then
rm $COOKIE_FILE_LOCATION
exit
else

## PRINT THE FILE TO DOWNLOAD ##
echo "File to download: https://${INSTANCE}/plugins/servlet/${FILE_NAME}"

curl -s -L --cookie $COOKIE_FILE_LOCATION "https://${INSTANCE}/plugins/servlet/${FILE_NAME}" -o "$LOCATION/JIRA-backup-${TODAY}.zip"



fi
rm $COOKIE_FILE_LOCATION

[Dario B]

Hi @CeDeAdmin and @Chris Dunne _Raledo_,

Not sure this is still needed, however, I have just tested the below script to backup Confluence Cloud and I can confirm it works fine:

#!/bin/bash

##---Fill-up the user account details, instance url and timezone-----
USERNAME=USER_NAME_HERE
PASSWORD=PASSWORD_HERE
INSTANCE=INSTANCE.atlassian.net
LOCATION="/path/to/download/folder"

# Set this to your Atlassian instance's timezone.
# See this for a list of possible values:
# https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE=Europe/Amsterdam
 
##----START-----###
echo "Starting the script..."

# Grabs cookies and generates the backup on the UI. ########
### PLEASE NOTICE THAT THE SESSION IS CREATED BY CALLING THE JIRA SESSION ENDPOINT!!! ######
#### THE SCRIPT DOES NOT WORK IF JIRA IS NOT INSTALLED !!! #######
TODAY=$(TZ=$TIMEZONE date +%Y%m%d)
COOKIE_FILE_LOCATION=jiracookie
curl --silent --cookie-jar $COOKIE_FILE_LOCATION -X POST "https://${INSTANCE}/rest/auth/1/session" -d "{\"username\": \"$USERNAME\", \"password\": \"$PASSWORD\"}" -H 'Content-Type: application/json' --output /dev/null
## The $BKPMSG variable will print the error message, you can use it if you're planning on sending an email
BKPMSG=$(curl -s --cookie $COOKIE_FILE_LOCATION --header "X-Atlassian-Token: no-check" -H "X-Requested-With: XMLHttpRequest" -H "Content-Type: application/json" -X POST https://${INSTANCE}/wiki/rest/obm/1.0/runbackup -d '{"cbAttachments":"true" }' )

 
## Checks if the backup procedure has failed
if [ "$(echo "$BKPMSG" | grep -ic backup)" -ne 0 ]; then
rm $COOKIE_FILE_LOCATION
echo "FAILED, IT RETURNED $BKPMSG"
exit
fi

## Checks if the backup exists every 10 seconds, 2000 times. If you have a bigger instance with a larger backup file you'll probably want to increase that.
for (( c=1; c<=2000; c++ ))
do
PROGRESS_JSON=$(curl -s --cookie $COOKIE_FILE_LOCATION https://${INSTANCE}/wiki/rest/obm/1.0/getprogress.json)
FILE_NAME=$(echo "$PROGRESS_JSON" | sed -n 's/.*"fileName"[ ]*:[ ]*"\([^"]*\).*/\1/p')

##ADDED: PRINT BACKUP STATUS INFO ##
echo "$PROGRESS_JSON"

if [[ $PROGRESS_JSON == *"error"* ]]; then
break
fi

if [ ! -z "$FILE_NAME" ]; then
break
fi
sleep 10
done

#If after 20 attempts it still fails it ends the script.
if [ -z "$FILE_NAME" ];
then
rm $COOKIE_FILE_LOCATION
exit
else

## PRINT THE FILE TO DOWNLOAD ##
echo "File to download: $FILE_NAME"

curl -s -L --cookie $COOKIE_FILE_LOCATION "https://${INSTANCE}/wiki/download/$FILE_NAME" -o "$LOCATION/CONF-backup-${TODAY}.zip"

fi
rm $COOKIE_FILE_LOCATION

Also this one is a Bash shell script but it is very clear and readable and it is exposing the correct endpoints to be called in order to automate Confluence backup.

I hope this helps.

[Chris Dunne _Raledo_]

Never mind, I figured out you can check this with the /rest/internal/2/task/progress/ API call

[Chris Dunne _Raledo_]

Hi Is there an api that will allow me get the details (mediaFileId and filename) of the last backup?

Thanks

Chris