Can't repo sync anymore

[tsubus]

Up until a couple of days ago (I guess before the LFS beta was available), I was able to repo sync my android tree with bitbucket, and had no issues replicating with gerrit to bitbucket. Ever since that change, I get this error when repo syncing:

channel 2: open failed: administratively prohibited: cannot open additional channels
mux_client_request_session: session request failed: Session open refused by peer

And something very similar with gerrit replication. I tried temporarily switching my repos to another git server provider, and as I expected everything is ok. Any info on this? Thanks.

[potiuk · Accepted solution]

I struggled with this for a while but had no time to take a closer look. Now I know what happens and I have a  workaround.

The problem is that Bitbucket disabled capability of multiplexing for SSH connections. In short: multiplexing works in the way that you only connect to the same server once. All the other connections are executed as additional channels opened via the single master connection already opened. Git automatically runs ssh commands in multiplexing mode under the hood. You can see that if you run the following while repo sync is running:

ps -eaf | grep ssh

You will see something similar to:

ssh -o ControlMaster=no -o ControlPath=some_path_here .....

Unfortunately you cannot use .ssh/config (because -o parameters override it).

UPDATE: See below answer - apparently you can have ControlMaster set to yes in .ssh and it works fine 

The solution that I found was to change default SSH command that GIT uses and disable multiplexing altogether. You can set GIT_SSH_COMMAND as follows:

export GIT_SSH_COMMAND="ssh -o ControlPath=none"

Once you do it - git will use your ssh command instead of the default and it will open new SSH connection for every git command separately. It's a bit slower than when multiplexing works but when Bitbucket rejects multiplexed channels, repo sync is even slower (by default ssh will fall-back to non-multiplexing connection when the multiplexing fails so it still works despite the error messages - but much slower). 

Add it to your .bashrc or similar and you should be good to go.

One drawback of this solution is that it is a global setting - all your git via ssh will stop using multiplexing. 

[potiuk]

Update: It seems also that works (and it localised to Bitbucket only - so it's better solution):

In your .ssh/config add ControlMaster yes for bitbucket:

Host bitbucket.org
        ControlMaster yes

@Michael Bayer -> we also had problem with replication. Some repos from Gerrit failed to replicate to Bitbucket (especially when there were branches or deleted changes). I know it's a lame solution but we solved it in a simple way. Additionally to built-in Gerrit replication we also run a crontab job every 10 minutes to mirror our repos to Bitbucket - so most of the changes are replicated almost immediately by Gerrit, but those that fail are later re-synced using the external mirroring...

Here is the script that we use:

#!/bin/bash
date
pushd /mnt/gerrit/git
SKIPPED_REPOS="(All-Projects.git|All-Users.git)"
for repo in *
do
  echo
  if [[ $repo =~ ^$SKIPPED_REPOS$ ]]; then
    echo "Skipping mirroring $repo"
    continue
  fi
  if [ ! -d "$repo" ]; then
    echo "Skipping non-directory $repo"
    continue
  fi
  pushd $repo >/dev/null
  REPO_URL=git@bitbucket.org:someprefix/$repo
  echo "Mirroring $repo to $REPO_URL"
  git push --mirror $REPO_URL 2>&1
  popd >/dev/null
done
popd
date

It mirrors a bit more than the default gerrit replication (includes gerrit config as well for the repo) - but it's still OK.

It also increases traffic for Bitbucket quite significantly. I wonder if the latest SSH outages are connected to the fact that some people like us are generating lot more traffic for Atlassian than is needed because we are workarounding some not-well-thought changes on their side. 

Hey Atlassian !!! - Hear us out please. Enable the SSH mulitplexing back. It will save you ton of traffic and infrastructure to handle it.

[Bryan Turner]

I'm one of the Bitbucket Server developers. There's been some confusion regarding SSH multiplexing and Bitbucket Server and I want to try and offer some clarification here, since people are finding this question.

Bitbucket Cloud (bitbucket.org) has removed multiplexing support, but it has not been removed from Bitbucket Server. Bitbucket Cloud and Bitbucket Server are separate codebases, so the removal of multiplexing support in Cloud does not affect Server.

GerritForge Support has been sending out e-mails to warn administrators about removed multiplexing, and in their e-mails they're indicating it's been removed from both Bitbucket Cloud and Server. This is untrue; it's still present, unchanged, in Bitbucket Server.

bturner@ubuntu:/tmp$ netstat -an | grep 7999
tcp        0      0 2.2.2.2:57472   1.1.1.1:7999         ESTABLISHED
unix  2      [ ACC ]     STREAM     LISTENING     361518   /home/bturner/.ssh/mux/git@host:7999.b10WWju1c6jjJ4V
unix  3      [ ]         STREAM     CONNECTED     360565   /home/bturner/.ssh/mux/git@host:7999.b10WWju1c6jjJ4V
unix  3      [ ]         STREAM     CONNECTED     359845   /home/bturner/.ssh/mux/git@host:7999.b10WWju1c6jjJ4V

As shown, 3 concurrent Git clones using a single TCP/IP socket with multiplexing.

Best regards,
Bryan Turner
Atlassian Bitbucket

[potiuk]

Still it would be nice to hear from Bitbucket Cloud developers what are they going to do about removing SSH multiplexing support that it causing so many problems. What's the reasoning, why so suddenly after it worked? Any plans for bringing it back since many people have this problem?

[Mark Adams]

Hi @Jarek Potiuk,

The easiest way to reach out to myself and the other Bitbucket Cloud developers with suggestions or feedback on feature changes is through our public issue tracker: https://bitbucket.org/site/master/issues

Thanks!

Mark

[potiuk]

Hello @Mark Adams. I went to the issues page, but then it asked me instead of creating issue, to create a support ticket instead . Which I just did: https://support.atlassian.com/servicedesk/customer/portal/11/BBS-42335 Looking forward to swift resolution of that problem. The problem with our workarounds is that they really put a lot of stress on your infrastructure - We have a job that mirrors about 100 repos every 10 minutes just to werkaround the replication issue with gerrit. If another 100 companies picks up the workaround ....

[Jack Enneking]

Dear future people: Bitbucket Cloud said here:
https://web.archive.org/web/20180614213754/https://bitbucket.org/site/master/issues/13465/please-enable-ssh-multiplexing-back
that they removed it on purpose and won't add it back.

:_(

[Michael Bayer]

I am having this problem however it is with Gerrit, which is using Jsch for SSH access, not sure if it execs ssh or what however setting the above environment variable in /etc/default/gerritcodereview and restarting has no effect.

Can someone please provide workarounds for gerrit replication?  thanks.

[Mark Adams]

Hi Michael!

We've ran into a couple of customers experiencing problems with JSch and Bitbucket. A few months ago we deprecated some older cryptographic algorithms as part of some updates to improve SSH performance and security and that appears to have caused some problems with certain SSH configurations.

SSH clients and libraries connecting to Bitbucket need to support at least one of the algorithms from each of the following categories:

• Encryption: aes128-ctr, aes128-gcm@openssh.com, aes192-ctr, arcfour128, arcfour256
• Key exchange: curve25519-sha256@libssh.org, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521
• Message authentication (MAC): hmac-sha1, hmac-sha1-96, hmac-sha2-256

I'd recommend that you take a look at the version of JSch that is in use and verify that it supports the above algorithms and update if needed. If you are running an up-to-date version of JSch, you may want to verify that the library is being configured in such a way that the algorithms above are enabled and allowed to be used when connecting via SSH.

Please let us know if you have further questions.

Mark

[Michael Bayer]

Hi Mark -

I understand that the SSH protocol in Bitbucket has been altered.   However what I don't know is how to change anything with how jsch runs.  I just tried a full upgrade to Gerrit 2.13.1, which also has the same problem. jsch itself is bundled with gerrit inside of the .war file.    It seems that other users are basically posting shell scripts that bypass using gerrit replication entirely.   This is obviously not sufficient for my needs and I'll continue searching.

[potiuk]

Hello @Mark Adams - this is completely wrong answer. The problem is already investigated and we know the reason - and those are not algorithms or ciphers. The problem is that Bitbucket stopped supporting SSH multiplexing (see for example here for explanation of what SSH multiplexin is: http://blog.scottlowe.org/2015/12/11/using-ssh-multiplexing/).

Many tools relies on the fact that when there are multiple repos or branches at the same host, they can all be connected to the same host via single SSH connection that acts like master - all the other connection simply reuse the SSH connection (extra channels) and act as slaves. This is especially important where we build android OS - we have some 600 repositories where often one sync operation (using repo tool or Gerrit) is pulling/pushing data to 30-50 repositories at the same time. In such cases SSH multiplexing is real saver on the communication overhead (and much less stress on your infrastructure).

Some tools (like repo) are able to cope with this (fallback on failed channels to opening another SSH connection) but some others (like the Jsch implementation used in gerrit) simply stops replicating repository (permanently and silently - except the logs) if such error occurs. It's a huge problem.

So the root cause of the problem is deliberately disabling of SSH multiplexing by Bitbucket (it worked flawlessly before July 2016).

[Michael Bayer]

Hi Jarek -

yeah, I wanted to post this yesterday but "Atlassian answers" also wants to limit how many posts I can make per day until I earn some kind of "points".   Which is odd because I thought my $10/ month would count for at least a couple of these "points"   (to be fair it looks like the newer billing plan hasn't been billing me lately).

your script is pretty nice and also the folks on the gerrit list are telling me not to use the replication feature either which I find odd.

[tsubus]

Jarek's solution works, thanks!

[Rob Lee]

I am seeing the same problem, but apparently it isn't happening to everyone, so I don't know what the correlation is.  In my case I am using both an Ubuntu 15.10 and 14.04 environment.