Upgrading to new SSH host keys causes some strange errors

[Dave Ludwig]

After following this article (https://bitbucket.org/blog/ssh-host-key-changes) 2 of my users (so far) have experienced similar errors and although the operation as a whole seems successful, the errors leave me confused as to why. I will post the full text of the sequence below, essentially the same happened to both of them. 

Why is the users personal SSH key involved, but more importantly what could be causing the errors? Neither of them have any trouble using Git or Sourcetree.

I did examine the known_hosts file, it seems like the ssh-keygen operation was a success. But the final ssh command does not verify success.

In the case of this user, he only has 2 SSH files in that folder, id_rsa.pub and id_rsa.ppk.

Thank you.

$ ssh git@bitbucket.org host_key_info
load pubkey "/c/Users/johnb/.ssh/id_rsa": invalid format
The authenticity of host 'bitbucket.org (104.192.141.1)' can't be established.
ECDSA key fingerprint is SHA256:FC73VB6C4OQLSCrjEayhMp9UMxS97caD/Yyi2bhW/J0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'bitbucket.org,104.192.141.1' (ECDSA) to the list of known hosts.
Connection closed by 104.192.141.1 port 22

$ ssh-keygen -R bitbucket.org && curl https://bitbucket.org/site/ssh >> ~/.ssh/known_hosts
# Host bitbucket.org found: line 1
/c/Users/johnb/.ssh/known_hosts updated.
Original contents retained as /c/Users/johnb/.ssh/known_hosts.old
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1267 100 1267 0 0 8121 0 --:--:-- --:--:-- --:--:-- 8121

$ ssh git@bitbucket.org host_key_info
load pubkey "/c/Users/johnb/.ssh/id_rsa": invalid format
Warning: Permanently added the ECDSA host key for IP address '104.192.141.1' to the list of known hosts.
git@bitbucket.org: Permission denied (publickey).

[Theodora Boudale · Accepted solution]

Hi Dave,

There are different SSH clients available. The instructions in our blog apply to OpenSSH (or compatible) clients.

The extension .ppk in the SSH key (id_rsa.ppk) indicates that this user may be using the SSH client PuTTY instead, and not OpenSSH. PuTTY in Windows does not use the ~/.ssh/known_hosts file but stores the known hosts in Windows Registry.

You can ask your developers to look at the following FAQ, the questions related to PuTTY and SourceTree for Windows for instructions on how to update the known hosts for PuTTY:

• https://confluence.atlassian.com/bbkb/bitbucket-cloud-ssh-host-key-rotation-faq-1235851234.html

When the developers use the following command from terminal, it is possible that another SSH client is used (most likely OpenSSH) and the format of the PuTTY keys is not recognized by OpenSSH.

ssh git@bitbucket.org host_key_info

They can confirm which SSH client is used from terminal if they run the following command and observe the first line of the output.

ssh -Tv git@bitbucket.org

The StackOverflow post that Victor shared has some info on how to generate SSH keys that are compatible with OpenSSH from your existing ones. However, if the developers are only using SourceTree with PuTTY, this is not necessary.

Please feel free to reach out if you have any questions.

Kind regards,
Theodora

[Dave Ludwig]

Hi Theodora,

Thank you for your reply. I do have some clarification questions.

We are indeed using sourcetree/putty/pageant to manage our keys. As this is the default way to do so in sourcetree as far as I know. So this is likely the cause of the errors we were seeing. I did not see them because I personally also have some extra openssh keys I use, in the same folder.

How do I know the correct name/value pair to put in the Registry? I have sourcetree installed with a PPK key and my registry looks very different than what the simplified.guide link shows. Mine looks like this:

Value name: "rsa2@22:bitbucket.org", Value Data: "0x23,0xb9b88df[lots more characters]"

So where do I get the correct value name and data to enter, and is it the same for everyone at my company?

Why is he deleting the key he just added in the video? (In the screenshots, there are 2 keys but in the video only one)

Secondly, after that is complete, how do I verify that putty is using the new key? I do not see a method to do so in the putty GUI.

And is there a different fix required to use bitbucket.org directly in a web browser?

Thank you!

[Theodora Boudale]

Hi Dave,

Thank you for the info, the public key in the registry seems to be hashed indeed. We are looking into this and reviewing internally, I will get back to you with an answer.

Kind regards,
Theodora

[Theodora Boudale]

Hi Dave,

You can follow these steps:

1. Follow this guide in order to delete the existing fingerprint for bitbucket.org from the registry

How to edit known hosts key fingerprint for PuTTY

Why is he deleting the key he just added in the video? (In the screenshots, there are 2 keys but in the video only one)

I think he is simply demonstrating how to update a key, and then how to delete a key.

2. Open Git Bash and run the following command

"C:\Users\username\AppData\Local\SourceTree\app-3.4.12\tools\putty\plink.exe" git@bitbucket.org

Make sure to replace username with the name of your user on this computer, and 3.4.12 with the version of SourceTree you are using.

3. You will see a message similar to the following. Before entering y, please make sure that the fingerprint you see in the message matches one of the fingerprints listed on this page, in the section SSH Host Keys:

• Configure SSH and two-step verification

The host key is not cached for this server: 
bitbucket.org (port 22)
You have no guarantee that the server is the computer
you think it is.
The server's ssh-ed25519 key fingerprint is: 
ssh-ed25519 255 SHA256:ybgmFkzwOSotHTHLJgHO0QN8L0xErw6vd0VhFA9m3SM
If you trust this host, enter "y" to add the key to
PuTTY's cache and carry on connecting.
If you want to carry on connecting just once, without
adding the key to the cache, enter "n".
If you do not trust this host, press Return to abandon the
connection.
Store key in cache? (y/n, Return cancels connection, i for more info)

4. When you confirm that it does, enter y.

You should now see a new entry in the registry with the new key.

When I tested this, PuTTY chose to use Bitbucket's Ed25519 key (as you can see from the warning I shared). I don't know if older versions of PuTTY will give precedence to the rsa key. In such a case, you would need to perform these steps again on 20th June after 17:00 UTC, when we will replace our current RSA host key with the following. However, if you see that PuTTY chooses either the Ed25519 key or the ECDSA key, then you won't need to take any further action.

I have discussed this with my team and we will update the FAQs to provide the info I shared with you.

Secondly, after that is complete, how do I verify that putty is using the new key? I do not see a method to do so in the putty GUI.

I am not aware of a way to check that in PuTTY GUI, the only way to check that I know of is by checking what key is present in the registry.

And is there a different fix required to use bitbucket.org directly in a web browser?

The change we made doesn't affect accessing Bitbucket Cloud from browser. It only affects Git operations over SSH (like clone, pull, push over SSH).

Please feel free to reach out if you need anything further.

Kind regards,
Theodora

[Dave Ludwig]

Hi Theodora,

After I say 'y' to store the key in putty's cache, the next lines in the cmd window are this:

Using username "git".
FATAL ERROR: No supported authentication methods available (server sent: publickey)

Thank you,

Dave

[Dave Ludwig]

Theodora, I think I fixed it. I went into Sourcetree options and found I had set the SSH agent to OpenSSH when messing around. Set it back to Putty/Plink and then the command worked. Boy its hard to figure out what relates to what! Thank you for your help. I will post if I see anything else.

[Theodora Boudale]

Hi Dave,

Thank you for the update, it's good to hear that it worked!

Please feel free to reach out if you need anything else.

Kind regards,
Theodora

[Dave Ludwig]

Please tell me this is normal to see still, after having done the upgrade... :-)

Thank you!

[Theodora Boudale]

Hi Dave,

I believe this message will show on every push that uses SSH, I'll double-check with the developers and get back to you.

Kind regards,
Theodora

[Theodora Boudale]

Hi Dave,

I got confirmation from the development team that this warning was showing to all users pushing via SSH. The warning has now been disabled.

Kind regards,
Theodora

[Victor Menes]

Hey Dave,

Please let me add a few guidelines that I hope are helpful for you getting this sorted out :)

Regarding the "Permission denied (publickey)", have you already checked Cloud's permissions denied resolutions ?

I also would think that taking a look at that to understand the formats related to the error "invalid format" it is also being thrown on the error you sent.

I hope this is helpful somehow to get it sorted out, Dave! :)