OpenSSH 8.8 client incompatibility and workaround

[#1]

Update (2021-10-21)

Bitbucket Cloud now supports rsa-sha2-256 and rsa-sha2-512 algorithms. The OpenSSH 8.8 client will function without the need for a workaround.

The team deployed a fix on Tuesday, Oct 19. After monitoring for two days we are confident that the OpenSSH 8.8 incompatibility has been resolved.

---

The latest release of OpenSSH — version 8.8, released on September 26th — introduced a configuration change that prevents that client from connecting to Bitbucket Cloud over SSH. Bitbucket engineers are actively addressing this and there are workarounds available in the meantime.

Problem

Newer SSH clients (OpenSSH >= 8.8) may be unable to connect to Bitbucket Cloud using SSH. Affected users will receive the following error message:

Unable to negotiate with <ip address> port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss

Default configurations of some newer SSH clients have dropped support for older signature algorithms used during the authentication process. Since Bitbucket Cloud’s SSH server does not yet support the newer preferred algorithms (rsa-sha2-256 and rsa-sha2-512), affected SSH clients are unable to complete authentication and an error is displayed.

Bitbucket Cloud engineers are actively in the process of adding support for these algorithms as a part of the SSH connection process.

Workarounds

If you are receiving the warning above, there are two options: updating your SSH config locally, or switching from SSH to HTTPS.

Update local SSH configuration

You can continue to use SSH by adding the lines below into the Host bitbucket.org section of your SSH configuration:

On Unix-like systems, this configuration is located at $HOME/.ssh/config or /etc/ssh/ssh_config.

On Windows systems, this configuration is located at %USERPROFILE%\.ssh\config or %PROGRAMFILES%\Git\etc\ssh\ssh_config.

Host bitbucket.org
    HostkeyAlgorithms +ssh-rsa
    PubkeyAcceptedAlgorithms +ssh-rsa

Use HTTPS instead of SSH

HTTPS connections to Bitbucket Cloud are unaffected by changes to the OpenSSH client. Therefore, you can avoid this issue by updating your git client to use HTTPS instead of SSH to talk to Bitbucket Cloud by following the instructions on this page. Switching to HTTPS will require using a different authentication mechanism. We recommend using an app password for automated git clients such as build machines or if you have two-factor authentication enabled.

To remove this workaround in the future, follow the same instructions to change your remote URL back to the SSH URL.

Next steps

The Bitbucket team is working on adding the updated signature support to our SSH server and we will provide another update once these changes have been implemented, tested, and deployed to our production infrastructure. Once we have provided that update, you may safely revert the changes you made per the instructions in this article. Follow this Community post or watch the BCLOUD ticket to be notified.

Questions about SSH workaround

Is my SSH connection to Bitbucket Cloud still secure using this workaround?

Yes, your SSH connection to Bitbucket Cloud is still secure. While ssh-rsa is insecure when signing arbitrary (i.e. attacker-controlled) data, that doesn’t help an attacker in this case. In the context of an SSH handshake with Bitbucket Cloud, ssh-rsa signature algorithm does not present a vulnerability that can be exploited to decrypt traffic or impersonate another user.

What’s a signing algorithm and how is it used?

A signing algorithm generates a secure signature that validates both the data being signed and the identity of the signer. SSH uses signatures to verify both host and client identities.

Why did OpenSSH drop support for ssh-rsa as an acceptable signing algorithm?

OpenSSH is removing support for older signature algorithms that may become less secure in the future. Details of the deprecation are in the Release Notes for OpenSSH 8.7.

Why is a workaround necessary?

Bitbucket Cloud’s SSH server implementation does not yet support the replacement signature algorithms - rsa-sha2-256 and rsa-sha2-512 - used in an SSH handshake.

Bitbucket engineers are actively working on adding this support - watch this BCLOUD ticket to be notified.

What are the HostkeyAlgorithms and PubkeyAcceptedAlgorithms settings?

These settings control what signatures algorithms the client will accept as valid from the host and client to demonstrate ownership of their private keys, respectively. OpenSSH 8.8 removes ssh-rsa as a default value in these settings.

Where exactly does ssh-rsa get used?

Bitbucket Cloud supports ssh-rsa in two contexts: signing the host key and signing client keys.

For host keys, ssh-rsa signs the result of the key exchange algorithm, which is a hash (used as the session identifier) of both client-provided and server-provided data along with the shared secret from the key exchange itself. An attacker cannot control this hash without breaking the key exchange algorithm. The key signing algorithm is not vulnerable to things like chosen prefix attacks because it hashes and signs trusted input, potentially using different hashes at different steps

RFC 4253 explicitly calls out this hash-of-hash behavior in Section 8.

For client keys, ssh-rsa signs the session identifier with extra data appended. This extra data is provided by the client, but is constrained by the server: it must match a known user and key. Since this data is controlled by the host, it cannot be used by an attacker to exploit a vulnerability in the key signing algorithm.

[Tobias Richter]

Is there a release date or a planned patch? 
How does the solution will look like, do I have to renew all my keys to a different algorithm. 

Please let us know, to plan the next ssh maintenance session.  :)

[Tom Kane]

Bitbucket Cloud now supports rsa-sha2-256 and rsa-sha2-512 algorithms. The OpenSSH 8.8 client will function without the need for a workaround.

The team deployed a fix on Tuesday, Oct 19. After monitoring for two days we are confident that the OpenSSH 8.8 incompatibility has been resolved.

[Sachin gupta]

Hi @Tom Kane 

Is there any fix for datacenter version. We are having same issue with datacenter version.

[Andres]

Hi @Tom Kane 

we are having the same problem for our version of datacenter, is there any documentation or update to solve the problem?

[Tobias Richter]

@Andres @Sachin gupta The issue is to be settled in the SSH system of your server, if you run a older SSH version the clients have to use a SSH client <8.8 

Find out if the SSH is apart of the Bitbucket Server or the OS System, if Bitbucket uses the e.g. Linux Env you can do it by your own. 

If it is a Bitbucket internal SSH Service, the only good way is to open a support ticket, and ask for a patch. This way the Atlassian Friends get a Ticket they have to care about ... Jira works you know ^.^