Bitbucket Server Rest Api: Clone all repositories

[SW]

I want to clone all repositories in my Bitbucket Server in order to have backups.

I trying to use Bitbucket rest api but not getting the required list of all repositories.

  curl -u username:password  https://servername:9090/rest/api/1.0/projects/~username/repos -k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    61    0    61    0     0    230      0 --:--:-- --:--:-- --:--:--   230{"size":0,"limit":25,"isLastPage":true,"values":[],"start":0}

 Is anything I am missing in command.

I have gone through the Bitbucket rest api doc but and using same command from there but not getting the required result.

[shawn_woodford]

Try this curl:

curl -sku username:password https://servername:9090/rest/api/1.0/users/usernamehere/repos

[Salim Ben Allal]

Is there a way to move from Stash to Bitbucket cloud?

[Firstborn]

Yes using the Stash server API and Bitbucket Cloud API but you have to do everything yourself manually or with a script, there is no Atlassian tool and they won't help you with the migration.

[Salim Ben Allal]

So I guess for a few repositories, better to this manually. It is SHAME that atlassian does not provide a tool that allow the migration between Atlassian products

[shawn_woodford]

Yes I agree, I developed a bash script to handle a migration which I may release on GitHub if I can clean it up and make it easy for others to run.

[Salim Ben Allal]

That would be great and I am sure it will help a lot of people :-)

[shawn_woodford]

Hi @Salim Ben Allal I just released my migration script which works on a Linux Bitbucket Server: https://github.com/swoodford/migrate_bitbucket_server_to_bitbucket_cloud

[Stephanie]

Hello Community!

I'm a designer on Bitbucket and we are doing further research on this topic and would love to invite you to take part in an upcoming customer research study.

Who we’re interested in talking to:

• Admins who have recently migrated or considered migrating

What’s involved in the research:

• Sessions are 1 hour and conducted over video-conference, so you can participate from anywhere around the globe.

• During the research, we'll start with a general chat to get to know you, then, we’ll have some questions to guide the conversations.

• As a token of our appreciation, you'll receive an e-gift card worth $100 USD within 5 days of completing your session.

If you're interested in taking part, fill out the form here.

If you have any other questions at all, feel free to reply to this post. We look forward to meeting you!

Cheers,

Stephanie

[Stephanie]

Hello Community!

I'm a designer on Bitbucket and we are doing further research on this topic and would love to invite you to take part in an upcoming customer research study.

Who we’re interested in talking to:

• Admins who have recently migrated or considered migrating

What’s involved in the research:

• Sessions are 1 hour and conducted over video-conference, so you can participate from anywhere around the globe.

• During the research, we'll start with a general chat to get to know you, then, we’ll have some questions to guide the conversations.

• As a token of our appreciation, you'll receive an e-gift card worth $100 USD within 5 days of completing your session.

If you're interested in taking part, fill out the form here.

If you have any other questions at all, feel free to reply to this post. We look forward to meeting you!

Cheers,

Stephanie

[SW]

That is nice Edwin. thanks for the answer. Just wondering, how we can clone all stuff (each branch and tag).

[shawn_woodford]

My script will do this, it only requires using an admin level account to access the Bitbucket server

[SW]

Great ! I will have look, thank you. FYI, I am on windows environment.  

[edwin]

Try this. You may have to do some modifications.

Install stashy - https://pypi.python.org/pypi/stashy

#!/usr/bin/env python

import stashy, os
stash = stashy.connect("http://localhost:7990/stash", "admin", "admin")

projects = stash.projects.list()

for project in projects:
     for repo in stash.projects["%s" %(project["key"])].repos.list():
         for url in repo["links"]["clone"]:
             // http or ssh
             if (url["name"] == "ssh"):
                os.system("git clone %s" %(url["href"]))

[SW]

Thanks for the response edwin. I have alredy tried this and it will not work.

reasons:

1st- I am using bitbucket server.

2nd, With rest api of bitbucket server you can only list the personal repositories.

[edwin]

Try stashy instead (i've edited the script above), make sure you have access to the repos you want to backup.

[Suresh Sakhare]

While executing above query I got this error: 'bitbucket' is not defined 

[edwin]

@Suresh Sakhare

for repo in stash.projects["%s" %(project["key"])].repos.list():

[Suresh Sakhare]

Instead of bitbucket you can use stash

[edwin]

After the rename to Bitbucket is when I used:

bitbucket = stashy.connect()

[Suresh Sakhare]

@edwin I downloaded all repos from server, now I am trying to push them to cloud bitbucket. Is there any way to push those downloaded repositories to bitbucket cloud instance with the branches and pull request history?

Thanks in advance :)

[edwin]

@Suresh Sakhare,

It will be a continuation of the current script. Basically you can join the script to do the cloning from server and push to cloud.

For each project - create it in cloud and the same for each repo

Please look into 

Repo.create()

Not sure but there might be a way of passing in the project to create the repo under.

https://stashy.readthedocs.io/en/latest/api.html#stashy.repos.Repos.create

pwd = os.getcwd()
for project in projects:    
    cloud_bitbucket.projects.create(project["key"], project["name"]) 
    for repo in bitbucket.projects["%s" %(project["key"])].repos.list():         
        cloud_bitbucket.Repo.create() 
        for url in repo["links"]["clone"]:
             // http or ssh
             if (url["name"] == "ssh"):
                os.system("git clone %s" %(url["href"]))
                os.chdir('%s.git' %(repo["name"]))                    
                new_url = "ssh://git@bitbucket.org/<company>/%s.git" %(repo["name"])   
                os.system("git remote add new_url %s" %(new_url))                    
                os.system("git push -f --all new_url")                   
                os.chdir(pwd)

[shawn_woodford]

Hi @Suresh Sakhare, @SW, I just released my migration script which works on a Linux Bitbucket Server: https://github.com/swoodford/migrate_bitbucket_server_to_bitbucket_cloud

[Suresh Sakhare]

Thanks a lot@shawn_woodford, @edwin , It helps me.

Now I am working on pull-requets migration script, Doing the following steps.

Accessing pull requests history from server using server api.

1. Get all projects
   1. Get repositories from project.
      1. Get PullRequests from repo.
         1. Get PullRequest from PullRequests
         2. Create PR for cloud instance with server PR details.
         3. Based on the Server PR state update the cloud PR state.
      2.  
   2.  

Is there any need to create temp branch and do cherry-pickup and then raise PR?

Also facing issue to setup the Author while raising PR. It always creator of PR as author even I mention the author detail in PR data.

PR Data

{
 
 "title": "POP-1: Merged this PR",
 "description": "* POP-1: commit 1 merged\r\n* POP-1: commit 2 merged",
 "state": "OPEN",
 "open": true,
 "closed": true,
 "createdDate": 1535637935000,
 "updatedDate": 1535637945000,
 "source": {
 "id": "refs/heads/POP-2",
 "displayId": "POP-2",
 
 "repository": {
 "slug": "mergedpr1",
 "id": 1,
 "name": "mergedpr1",
 "scmId": "git",
 "state": "AVAILABLE",
 "statusMessage": "Available",
 "forkable": true,
 "public": false
 },
 "branch":{ 
 "name":"POP-2"
 }
 },
 "destination": {
 "id": "refs/heads/master",
 "displayId": "master",
 
 "repository": {
 "slug": "mergedpr1",
 "id": 1,
 "name": "mergedpr1",
 "scmId": "git",
 "state": "AVAILABLE",
 "statusMessage": "Available",
 "forkable": true,
 "public": false
 },
 "branch":{ 
 "name":"master"
 }
 },
 "locked": false,
 "author": {
 "type": "author",
 "user": {
 "username": "sureshsakhare",
 "displayName": "suresh sakhare",
 "type":"user"
 }
 
 },
 "reviewers": [
 {
 
 "username": "xyz",
 "emailAddress": "xyz.pqr@www.com",
 "displayName": "xyz pqr"

 }
 ],
 "participants": []
}

any suggestion on it ?

[shawn_woodford]

Great!  In my case I had never considered migrating PRs, but that all open PRs would be merged and closed before migration, so I have no experience creating a PR or migrating an open PR with the API.  I'd have to see how this could work.

[Suresh Sakhare]

Hi @shawn_woodford, I tried your script to clone and push repos from sever to cloud. It always return me the token expire error. I think this issue because of callback url field. If I am correct then what will be the value of callback url field.

{
 "error": {
 "message": "Access token expired. Use your refresh token to obtain a new access token."
 },
 "type": "error"
}

[Suresh Sakhare]

@shawn_woodford Its callback URL issue. I must set callback URL when using Oauth2 authentication.

Is there any need of ssh key? I am executing the this script with https protocal, In migrating repo to cloud step, I got the permission denied(publickey) issue.

[shawn_woodford]

Yes, you should follow the instructions from Bitbucket for creating your Oauth2 access, but the callback URL can be anything really since the script won't be using it for anything.

And yes, you do need to be able to run a `git clone` on all repos being migrated as the `root` user on the server that is running the script, so for this you can add an authorized ssh key to your Bitbucket Server with admin or system admin level permission, and make sure the root user can access that ssh key.  Save it in `/root/.ssh/id_rsa` and see lines 216-223, where it says `## If issues with SSH/git clone:`, just uncomment that block if you're still having trouble.