Up until a couple of days ago (I guess before the LFS beta was available), I was able to repo sync my android tree with bitbucket, and had no issues replicating with gerrit to bitbucket. Ever since that change, I get this error when repo syncing:
channel 2: open failed: administratively prohibited: cannot open additional channels
mux_client_request_session: session request failed: Session open refused by peer
And something very similar with gerrit replication. I tried temporarily switching my repos to another git server provider, and as I expected everything is ok. Any info on this? Thanks.
I struggled with this for a while but had no time to take a closer look. Now I know what happens and I have a workaround.
The problem is that Bitbucket disabled capability of multiplexing for SSH connections. In short: multiplexing works in the way that you only connect to the same server once. All the other connections are executed as additional channels opened via the single master connection already opened. Git automatically runs ssh commands in multiplexing mode under the hood. You can see that if you run the following while repo sync is running:
ps -eaf | grep ssh
You will see something similar to:
ssh -o ControlMaster=no -o ControlPath=some_path_here .....
Unfortunately you cannot use .ssh/config (because -o parameters override it).
UPDATE: See below answer - apparently you can have ControlMaster set to yes in .ssh and it works fine
The solution that I found was to change default SSH command that GIT uses and disable multiplexing altogether. You can set GIT_SSH_COMMAND as follows:
export GIT_SSH_COMMAND="ssh -o ControlPath=none"
Once you do it - git will use your ssh command instead of the default and it will open new SSH connection for every git command separately. It's a bit slower than when multiplexing works but when Bitbucket rejects multiplexed channels, repo sync is even slower (by default ssh will fall-back to non-multiplexing connection when the multiplexing fails so it still works despite the error messages - but much slower).
Add it to your .bashrc or similar and you should be good to go.
One drawback of this solution is that it is a global setting - all your git via ssh will stop using multiplexing.
Update: It seems also that works (and it localised to Bitbucket only - so it's better solution):
In your .ssh/config add ControlMaster yes for bitbucket:
Host bitbucket.org ControlMaster yes
#!/bin/bash date pushd /mnt/gerrit/git SKIPPED_REPOS="(All-Projects.git|All-Users.git)" for repo in * do echo if [[ $repo =~ ^$SKIPPED_REPOS$ ]]; then echo "Skipping mirroring $repo" continue fi if [ ! -d "$repo" ]; then echo "Skipping non-directory $repo" continue fi pushd $repo >/dev/null REPO_URLemail@example.com:someprefix/$repo echo "Mirroring $repo to $REPO_URL" git push --mirror $REPO_URL 2>&1 popd >/dev/null done popd date
It mirrors a bit more than the default gerrit replication (includes gerrit config as well for the repo) - but it's still OK.
It also increases traffic for Bitbucket quite significantly. I wonder if the latest SSH outages are connected to the fact that some people like us are generating lot more traffic for Atlassian than is needed because we are workarounding some not-well-thought changes on their side.
Hey Atlassian !!! - Hear us out please. Enable the SSH mulitplexing back. It will save you ton of traffic and infrastructure to handle it.
I am having this problem however it is with Gerrit, which is using Jsch for SSH access, not sure if it execs ssh or what however setting the above environment variable in /etc/default/gerritcodereview and restarting has no effect.
Can someone please provide workarounds for gerrit replication? thanks.
We've ran into a couple of customers experiencing problems with JSch and Bitbucket. A few months ago we deprecated some older cryptographic algorithms as part of some updates to improve SSH performance and security and that appears to have caused some problems with certain SSH configurations.
SSH clients and libraries connecting to Bitbucket need to support at least one of the algorithms from each of the following categories:
I'd recommend that you take a look at the version of JSch that is in use and verify that it supports the above algorithms and update if needed. If you are running an up-to-date version of JSch, you may want to verify that the library is being configured in such a way that the algorithms above are enabled and allowed to be used when connecting via SSH.
Please let us know if you have further questions.
Hi Mark -
I understand that the SSH protocol in Bitbucket has been altered. However what I don't know is how to change anything with how jsch runs. I just tried a full upgrade to Gerrit 2.13.1, which also has the same problem. jsch itself is bundled with gerrit inside of the .war file. It seems that other users are basically posting shell scripts that bypass using gerrit replication entirely. This is obviously not sufficient for my needs and I'll continue searching.
Hello @Mark Adams - this is completely wrong answer. The problem is already investigated and we know the reason - and those are not algorithms or ciphers. The problem is that Bitbucket stopped supporting SSH multiplexing (see for example here for explanation of what SSH multiplexin is: http://blog.scottlowe.org/2015/12/11/using-ssh-multiplexing/).
Many tools relies on the fact that when there are multiple repos or branches at the same host, they can all be connected to the same host via single SSH connection that acts like master - all the other connection simply reuse the SSH connection (extra channels) and act as slaves. This is especially important where we build android OS - we have some 600 repositories where often one sync operation (using repo tool or Gerrit) is pulling/pushing data to 30-50 repositories at the same time. In such cases SSH multiplexing is real saver on the communication overhead (and much less stress on your infrastructure).
Some tools (like repo) are able to cope with this (fallback on failed channels to opening another SSH connection) but some others (like the Jsch implementation used in gerrit) simply stops replicating repository (permanently and silently - except the logs) if such error occurs. It's a huge problem.
So the root cause of the problem is deliberately disabling of SSH multiplexing by Bitbucket (it worked flawlessly before July 2016).
Hi Jarek -
yeah, I wanted to post this yesterday but "Atlassian answers" also wants to limit how many posts I can make per day until I earn some kind of "points". Which is odd because I thought my $10/ month would count for at least a couple of these "points" (to be fair it looks like the newer billing plan hasn't been billing me lately).
your script is pretty nice and also the folks on the gerrit list are telling me not to use the replication feature either which I find odd.
I'm one of the Bitbucket Server developers. There's been some confusion regarding SSH multiplexing and Bitbucket Server and I want to try and offer some clarification here, since people are finding this question.
Bitbucket Cloud (bitbucket.org) has removed multiplexing support, but it has not been removed from Bitbucket Server. Bitbucket Cloud and Bitbucket Server are separate codebases, so the removal of multiplexing support in Cloud does not affect Server.
GerritForge Support has been sending out e-mails to warn administrators about removed multiplexing, and in their e-mails they're indicating it's been removed from both Bitbucket Cloud and Server. This is untrue; it's still present, unchanged, in Bitbucket Server.
bturner@ubuntu:/tmp$ netstat -an | grep 7999 tcp 0 0 22.214.171.124:57472 126.96.36.199:7999 ESTABLISHED unix 2 [ ACC ] STREAM LISTENING 361518 /home/bturner/.ssh/mux/git@host:7999.b10WWju1c6jjJ4V unix 3 [ ] STREAM CONNECTED 360565 /home/bturner/.ssh/mux/git@host:7999.b10WWju1c6jjJ4V unix 3 [ ] STREAM CONNECTED 359845 /home/bturner/.ssh/mux/git@host:7999.b10WWju1c6jjJ4V
As shown, 3 concurrent Git clones using a single TCP/IP socket with multiplexing.
Still it would be nice to hear from Bitbucket Cloud developers what are they going to do about removing SSH multiplexing support that it causing so many problems. What's the reasoning, why so suddenly after it worked? Any plans for bringing it back since many people have this problem?
Hello @Mark Adams. I went to the issues page, but then it asked me instead of creating issue, to create a support ticket instead . Which I just did: https://support.atlassian.com/servicedesk/customer/portal/11/BBS-42335 Looking forward to swift resolution of that problem. The problem with our workarounds is that they really put a lot of stress on your infrastructure - We have a job that mirrors about 100 repos every 10 minutes just to werkaround the replication issue with gerrit. If another 100 companies picks up the workaround ....
This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.Read more
Bitbucket Pipelines helps me manage and automate a number of serverless deployments to AWS Lambda and this is how I do it. I'm building Node.js Lambda functions using node-lambda ...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs