Upgrade Bitbucket from 7.6.4 to 8.8.0 Unable to interact with bitbucket over SSH

[Walton McDonald]

Hello,

We have recently upgraded our bitbucket instance from 7.6.4 to 8.8.0, The upgrade went flawlessly, except any interaction with bitbucket over SSH spawn this error message.

git clone ssh://git@bitb:7999/test/test.git
Cloning into 'test'...
Connection closed by 155.157.133.94 port 7999
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Here is the verbose output:

GIT_SSH_COMMAND="ssh -vvv" git clone ssh://git@bitb:7999/test/test.git
Cloning into 'test'...
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
FIPS mode initialized
debug2: resolving "bitb" port 7999
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to bitb [155.157.133.94] port 7999.
debug1: Connection established.
debug1: identity file /home/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/.ssh/id_ed25519 type -1

debug1: key_load_public: No such file or directory
debug1: identity file /home/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version APACHE-SSHD-2.9.2
debug1: no match: APACHE-SSHD-2.9.2
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to bitb:7999 as 'git'
debug3: put_host_port: [bitb]:7999
debug3: hostkeys_foreach: reading file "/home/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com
debug2: MACs stoc: hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com
debug2: compression ctos: zlib@openssh.com,zlib,none
debug2: compression stoc: zlib@openssh.com,zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-hellman-group18-sha512,diffie-hellman-group17-sha512,diffie-hellman-group16-sha512,diffie-hellman-group15-sha512,diffie-hellman-group14-sha256,ext-info-s
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: kex: ecdh-sha2-nistp256 need=20 dh_need=20
debug1: kex: ecdh-sha2-nistp256 need=20 dh_need=20
debug3: send packet: type 30
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 155.157.133.94 port 7999
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This is occuring on two different non-identical servers. I have tried altering our sshd_config and alerting the MTU of the server to no avail. If anybody could provide insight that would be much appreciated.

What the two systems share.

System Specs:

NAME="Oracle Linux Server"
VERSION="7.9"

git version 2.39.2

jkd-17.0.2

[Walton McDonald]

For any one also suffering from this issue, our current version of Openssh was attemting to use a disabled KexAlgorithm in java 17. We were able to fix the issue by rolling back to java1.8.0_162. While a bandaid a better solution would be upgrading your openssh version to a version which supports stronger algorithms.

[Mia Paulin]

@Walton McDonald  Make sure you have the right authnetication methods and that your credentials are correct for both.  Also, did you create the public and private keys for the ssh to be used on both machines?  Its saying that you didn't create a public key for the edndpoint you are trying to access.  You have to generate both a public and a private key when you generate the keys.  One is stored on from the local/on-premise machine of wherever you are(private key), and the other needs to be one the endpoint that you are connecting to(the public key).  You need to add it to the authenitcation method on the endpoint/ your remote git(bitbucket) instance. 

[Mia Paulin]

@Walton McDonald   The easiest way to do it is by creating the keys using git and storing the private one on your local machine and the public one on bitbucket.  You need to download git and see the bitbucket documentation for ssh implementation(paraphrasing). Since you will be using git(the actual CLI), use the Linux commands(not mac or windows).  It's easier.  Then, save a copy of the keys where you can remember them and no one else can find them.  Save it somewhere.  Then, when you create your authentitcation method, make sure its ssh and add your public key in the field.  Note:  If you can't find the private key, you can't use your git on your local machine to send things to and from bitbucket(or wherever else has that public key).  And, when cloning the repository, make sure the protocol is ssh, not https.  Also, in order to use the ssh and create the ssh keys, you need version of git.  Just google it(or use whatever browser you have) and downlpoad it acording to your OS on your on-premise computer.  The directions in the documentation show you how to create your keys through an ssh agent.  Also(this will help a lot), to to the Atlassian MarketPlace and download SourceTree.  And, someone had to help me with all of this so I'm helping you with information that I was helped with.  I hope this helps.

[Mia Paulin]

@Walton McDonald   Another thing people at Atlassian helped me with that I did.  This is important: build a pipeline using Bitbucket.  There is an abstraction of one, but it helps to customize it to suit your needs(as far as your application is concerned).  A lot of people at Atlassian helped me out and pointed me in the right dirction, so don't be afraid to ask.  People are more than willing to go that extra mile to help out a fellow Atlassian.  I've had a lot of help, trust me.