What to do with your Mercurial repos when Bitbucket sunsets support

[himeldas]

Guys the solution is simple. Just switch to GitHub, they have an import tool that can convert a hg repo into a git repo within minutes (all commits preserved). Hg was the main reason I chose bitbucket over GitHub years ago. Since I'm forced to use git now, might as well use github moving forward.

[philipstarkey]

@himeldas you may want to read the previous page as to why "just using the GitHub importer" is not as simple as it seems.

Spoiler: it is not guaranteed to preserve all commits for example. But there are ways to move to GitHub that do ensure you won't lose anything (see previous page).

[Klaus Rennecke]

The reason why a git importer cannot be complete is that the branching model is not equivalent. Git loses ancestry of any branch that gets merged one or more times.

And there lies the problem: You cannot move history to git without losing information. It is better than CVS but don't try to sell it for what it cannot deliver.

I was looking for a small service that can carry my personal projects and ended up here https://hg.sr.ht/. However, for production use I am looking at offline backups.

[kalthad]

@Klaus Rennecke:

an alternative is to use the hg-git plugin, thanks to the latest patch provided by Manuel Jacob

named branches are now supported (they are correctly translated to git branches)

[CharlieC]

FWIW, I'm probably moving my open source stuff to Heptapod who've kindly setup a project that will allow me to have several repos together. The initial import (commits, issues, PRs, etc.) went well and I was able to setup CI fairly quickly. The GitLab model is a bit different to Bitbucket, et al. as it doesn't support personal forks but I think it's worth giving it a go.

Atlassian is still extremely keen on force-feeding changes on us so I wonder what they'll break next. :-/

[Jerry Gardner]

Here's an issue we're seeing when converting our BitBucket Hg repositories to Git and pushing them back up to BitBucket: the repository when converted to Git exceeds the 2GB limit that BitBucket imposes...

How have others handled this situation? As far as I know, there's no way to get BitBucket to relax the repository size limitation, so we need a solution that's usable.

One thought is to limit how far back in time the commit history of the repository goes. For example, if the head of our respository is currently at v12.3.4, perhaps we can create the new Git repo containing history going back only to, say, v6.1.2.

Any thoughts?

[bressler]

I know that some of you are freaking out that your hg repos are marked for death. I have been freaked out for a while, but finally converted all my hg repos to git and I figured this might help someone.

Throwing this out there. NO SUPPORT. And apolgies in advance for how crap of a shell scripter I am. ALSO: I use both hg and git in the simplest way. No branches, no spider merges, one contributor, private repos. So if you aren't me, no guarantees.

Worked for me on Windows 10 Ent, Py3 3.6.7, hg 5.0.2, git 2.18, in a git bash window.

1. Install hg-git plug into hg following steps 1 and 2 of this article.

2. For any given repo (example repo here called "orbit") this is the basic procedure:

rm -rf ./orbit_git
rm -rf ./orbit_bare
mkdir ./orbit_bare
git init --bare ./orbit_bare
hg clone hg clone ssh://hg@bitbucket.org/<you>/orbit ./orbit
cd orbit
hg bookmark -r default master
hg push ../orbit_bare
cd ../orbit_bare && git clone . ../orbit_git
cd ../orbit_git
git remote set-url origin git@bitbucket.org:<you>/orbit_git.git
<create new git repo (named *_git) in BB, then>
git push
rm -rf orbit_bare

You don't have to create the dest repos in advance. You can make them after. But you DO have to git push the repo into the git repo (as shown above). All my commits and history came across, which to me is approximately magic, but hey that's great.

So given that, I hacked this shell script together.

IMPORTANT: I DID NOT WRITE THIS SCRIPT FOR YOU. YOU WILL NEED TO MODIFY IT.

IMPORTANT: No support. Lots of rm -rfs in there. so use at your own risk. Good luck.

#!/bin/bash

# hg -> git conversion script.
# This is loosely based on this article:
# https://www.markheath.net/post/how-to-convert-mercurial-repository-to

# For testing with just one repo
#repos=(
#  "myrepo1"
#)

# And once that works, let her rip on all your repos
repos=(
#  "repo1"
#  "repo2"
#  "repo3"          # if you've already run some, just comment them out
  "repo4"
  "best_repo"
  "love_this_one"
)

# For coloring output
r="\e[91m"
g="\e[92m"
b="\e[94m"
c="\e[96m"
m="\e[95m"
y="\e[93m"
d="\e[39m"

# This kinda works...to preflight what's going to happen without destroying
# everything, you can uncomment the pf=echo line and most destructive operations
# will echo the command rather than execute it.
#pf=echo
pf=

# IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT
# MODIFY THESE LINES
hgprefix="ssh://hg@bitbucket.org/<your_bb_user_name>/"
gitprefix="git@bitbucket.org:<your_bb_user_name>/"

echo -e "${m}==============================================================${d}"
echo -e "conv: 1.0"
echo -e "hg:   $(hg --version -q)"
echo -e "git:  $(git --version)"
echo -e "${m}==============================================================${d}"
echo -e

# Main loop
count=${#repos[@]}
for (( i=0; i<${count}; i++ ))
do
  repo=${repos[${i}]}
  hgurl="${hgprefix}${repo}"
  gitrepo="${repo}_git"
  giturl="${gitprefix}${gitrepo}"

  hgdir="${repo}"
  baredir="${repo}_bare"
  gitdir="${repo}_git"

  # Report
  fi=$((i+1))
  echo -e "${m}▒▓█■${d} ${fi}/${count}: ${c}${repo}${d} from ${hgurl} to ${giturl}:"

 # CWD: conv
  # Clone the hg repo
  if [[ -d "${hgdir}" ]]; then
    echo -e "hg dir already exists for ${repo}, skipping clone"
 else
   echo -e "  ${c}CLONING${d}"
  if ! ${pf} hg clone "${hgurl}" "./${hgdir}"; then
   echo -e "hg clone failed ${hgurl}"
   exit 1
  fi
 fi

 # CWD: conv
 # Init a new bare git repo to clone into
 if [[ -d "./${baredir}" ]]; then
   echo -e "  ${c}NUKING BAREDIR${d}"
  ${pf} rm -rf "./${baredir}"
 fi
 echo -e "  ${c}GIT INITING BAREDIR${d}"
 if ! ${pf} git init --bare "./${baredir}"; then
   echo -e "git init --bare failed for ${baredir}"
   exit 1
 fi

 # CWD: conv
 # Bookmark master
 pushd "./${hgdir}" &> /dev/null || exit 1
 # CWD: hgdir
 echo -e "  ${c}HG BOOKMARKNG HGDIR${d}"
 if ! ${pf} hg bookmark -r default master; then
   echo -e "hg bookmark failed for ${repo}"
   exit 1
 fi

 # CWD: hgdir
 # Push our repo into the bare repo. This magical operation is actually carried
 # out by the hg-git Mercurial plugin I have installed.
 echo -e "  ${c}HG PUSHING TO BAREDIR${d}"
  pwd
 if ! ${pf} hg push "../${baredir}"; then
  echo -e "hg push to bare repo failed for ${repo}"
  exit 1
 fi
 popd &> /dev/null || exit 1

 # CWD: conv
 # Now git clone the bare git repo into a regular git repo (I have *no idea*
 # what any of# this bare repo shit means.)
 if [[ -d "./${gitdir}" ]]; then
  echo -e "  ${c}NUKING GITDIR${d}"
  ${pf} rm -rf "./${gitdir}"
 fi
 pushd "./${baredir}" &> /dev/null || exit 1
 if ! ${pf} git clone . "../${gitdir}"; then
 echo -e "  ${c}HG CLONING TO GITDIR${d}"
   echo -e "git clone from bare to regular failed for ${repo}"
   exit 1
 fi
 popd &> /dev/null || exit 1

 # CWD: conv
 # Set remote origin on gitdir since we know what it will be called. We try to
 # push, but the remote may not exist yet
 pushd "./${gitdir}" &> /dev/null || exit 1
 echo -e "  ${c}SETTING REMOTE ORIGIN FOR GITDIR${d}"
 if ! ${pf} git remote set-url origin "${gitprefix}${gitdir}.git"; then
  echo -e "git remote set-url origin failed for ${gitdir}"
  exit 1;
 fi
 echo -e "  ${c}PUSHING GITDIR${d}"
 if ! ${pf} git push; then
   echo -e "git push failed for ${gitdir}, remote may not yet exist, ignoring."
 fi
 popd &> /dev/null || exit 1

 # CWD: conv
 # Delete intermediate bare repo
 echo -e "  ${c}NUKING BAREDIR${d}"
 if ! ${pf} rm -rf "./${baredir}"; then
  echo -e "Couldn't delete ${baredir}, ignoring"
 fi

 # Report
 echo -e "Conversion of ${repo}: ${g}SUCCESS${d}" 

[Zoltán Lehóczky]

@Jerry Gardner the solution to this is to move to GitHub instead. I'm not trying to be witty, really, in the long run you're much better off with GitHub, including larger repo sizes. GitHub's offering is another league compared to Bitbucket, even its free tier. If you have the time to do this right, I suggest you to move to GitHub, as we also did.

[kalthad]

@bressler 

hi

I have done this conversion in the past and I suggest a slightly different strategy (supposing that you are using named branches in your mercurial repository):
1.install the latest hg-git, one which supports, thanks to Manuel Jacob, named branches.
   then put the following 

2. in your gobal .hgrc

[git]
export-additional-refs.named-branch-heads:revset = head() and not branch('re:\Adefault\Z')
export-additional-refs.named-branch-heads:template = refs/heads/{branch}

3. make via the bitbucket interface a git repo (say your orginal repo is called mycool_code)
   call it mycool_code_git

4. hg-git does not support https only ssh, so make sure that your ssh public key is installed

5. in your local hgrc (of the mycool_code repo) add

bitbucketgit = git@bitbucket.org:userjoedoe/mycool_code_git.git

then run 

hg push bitbucketgit

thats it. the benefit is that other mercurial user can pull from that git repository and can 

still use named branches!

regards

Uwe Brauer

[bressler]

@kathad...that's great Uwe! Much simpler than my method.

-Bryan

[Gary]

@Jerry Gardner (and anyone in the same situation)please feel free to open a support ticket at https://support.atlassian.com/contact/#/, and my team can help you with the limit on those repositories.  

Thank you,

Gary Sackett
Bitbucket Cloud Support Lead

[bressler]

@Gary 

Did you guys delete my post from today with the shell script in it?

-Bryan

[Gary]

@bressler apologies for that - it was picked up by our spam filter.  Should re populate shortly.

Gary

[bressler]

Thank you @Gary !

[fritzophrenic_NA]

I didn't want to stop using Mercurial, and can't really do a paid service right now, so what I ended up doing was:

1. Import my repository into a (temporary) new GitHub repository
2. Use one of the Bitbucket-to-GitHub issue migrators to migrate the issues. I used https://github.com/jeffwidman/bitbucket-issue-migration but I discovered it does not migrate the attachments, so consider finding another if that will be a problem for you.
3. Create a new project on SourceForge
4. Use the SourceForge import tools to import the GitHub issues into Tickets in the SourceForge project.
5. Clone the empty SourceForge codebase onto my local machine. This is important for some reason!
6. "hg pull" my actual repository into the empty local repository cloned from SourceForge.
7. "hg push" back to SourceForge.
8. Add any missing attachments to the ticket system.

[Raphael Pepi]

Have been migrating hg to git repos within butbucket.  So far its been not so painful btu has invloved Tons of manual steps (creating the new repo,Moving across user access, ssh keys, jira links, pipelines, etc. ) 

The Issue Im facing is for forked repos.. the Fork information does not survive the migration process (using hg-fast-export).

So how to replicate the forks as they are? 

Does anyone have a solution for this? if so what is it?

Extremely Frustrating that Atlassian is not only dropping support, but also deleting your repo, while not providing a single helpfull automation tool even if it couldn't handle EVERY use case it could have at least got most of the way there...  

[philipstarkey]

@Raphael Pepi This would be a bit of a pain, but if you fork your new master Git repository on BitBucket, use hg-fast-export on an old hg fork, and then force push that converted fork to your new git fork on BitBucket, I think it might work?

It won't preserve issues/pull requests/etc though. My tool (bitbucket-hg-exporter - see earlier pages here), while designed for moving to GitHub, can create a static archive of that content without migrating to GitHub, so may be of interest (but there is no way to migrate PR discussions to your new git repos either on GitHub or BitBucket)

[eao197]

Can someone from Atlassian Team explain to me why I can't see my private Hg repos if the support for Hg will be removed only on 1 July? I can access public Hg repos, but don't have access to private ones. Do they already removed permanently?

**UPD.** It looks like a problem with authentication of users. After relogin I get the access to all repos.

[philipstarkey]

@eao197 I can still see my private Hg repos. I suggest you log an official support ticket with them as Atlassian staff have not been consistently responding here.

[Raphael Pepi]

how would one get a list of all the mercurial repos using the rest api?

the scripts ive found have either not worked, been based on 1.0 or just overall not worked

need this to make the task of backing up the 150+ mercurial repos a little easier.

Atlassian?

[Franz Rodenacker]

1. Set up an App password in your Bitbucket account with read rights

2. Call https://api.bitbucket.org/2.0/repositories/{bitbucketWorkspace}?page=1&pagelen=100 using basic authentication (your username & your app password); Change the page parameter to get to the second page
The result is a Json list of all your repos. The Mercurial ones are scm = hg

[Ruslan Berezyuk]

Ohh boy ... they still advertise they support mercurial ... 

[Klaus Rennecke]

That is certainly bad form. There are very nice Mercurial hosting services out there but Bitbucket is no longer one of them. Squatting on the ad-words is not about to change that.

The "likely to recommend..." score plummeted during the last months below zero.

[cja]

The easiest way to convert from Hg to Git seems to be to use the Github repository importer: https://github.com/new/import

This successfully converted and imported the first repository I tried, directly from Bitbucket. It asks for repository URL and Bitbucket user name and password and took about five minutes for the small repository I tested with.

Very easy to use. Shame that Bitbucket didn't make a similar tool. I was happy here but clearly Github wanted me more.

Once my repositories are in Github then I'll give it a try and if I don't like it I may go back to Bitbucket, or to Gitlab.

I'll be sad not to use Hg any more but I reckon I'll have fewer hosting problems if I switch to Git. Now I'll be looking for the equivalent of Joel's HgInit - https://web.archive.org/web/20180924182907/http://hginit.com/

[kalthad]

@cja  why don't you continue using hg, and the hg-git plugin, That works nice and supports even named branches

Uwe Brauer