Connection closed by remote host

[Alberto Samorì]

Hi Team,

we are experiencing a discontinuous error from our CI/CD Jenkins server (it's an old release: CentOS release 6.10) while we are trying to connect to the bitbucket repository, running for example a command like git fetch --tags --force --progress -- git@bitbucket.org:workspace-id/repo.git +refs/heads/:refs/remotes/origin/; the error is the following:

09:53:18 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- git@bitbucket.org:workspace-id/repo.git +refs/heads/:refs/remotes/origin/" returned status code 128:
09:53:18 stdout:
09:53:18 stderr: ssh_exchange_identification: Connection closed by remote host
09:53:18 fatal: Could not read from remote repository.
09:53:18
09:53:18 Please make sure you have the correct access rights
09:53:18 and the repository exists.
09:53:18
09:53:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
09:53:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
09:53:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
09:53:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
09:53:18 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
09:53:18 ... 11 more

Even when we try, from the same server, to run a command like ssh -vvv git@bitbucket.org host_key_info we receive, after some successes, the error ssh_exchange_identification: Connection closed by remote host.

Do you have any clue about this issue?

Thank you in advance,

Alberto

[Theodora Boudale]

Hi Alberto,

Could you perhaps provide the full output of ssh -Tvvv git@bitbucket.org for a successful operation and the full output of the same command for a failed operation (on the same machine where Jenkins is running)? I would like to check if there is anything in the outputs that may indicate what is causing this.

Kind regards,
Theodora

[Theodora Boudale]

Hi @Alberto Samorì,

I saw that we have a few other reports about the same error from other customers.

You can create a ticket with the support team, you will then be asked to provide some additional debug info that the engineering team can use to investigate this (the support ticket will be visible only to you and Atlassian staff).

You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. When you are asked to provide the workspace URL, please make sure you enter the URL of the workspace that is on a paid billing plan to proceed with ticket creation.

Just a heads up, I removed the workspace ID and repo name from your post to protect your privacy.

Kind regards,
Theodora

[Alberto Samorì]

Hi @Theodora Boudale ,

thank you for your feedback and sorry for the workspace ID and repo name, I forgot to remove them.

Below you can find both success and failure debug info:

SUCCESS:
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/waswc/.ssh/config
debug3: key names ok: [ecdsa-sha2-nistp256,ssh-rsa,ssh-dss]
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to bitbucket.org [104.192.141.1] port 22.
debug1: Connection established.
debug1: identity file /home/waswc/.ssh/identity type -1
debug1: identity file /home/waswc/.ssh/identity-cert type -1
debug3: Not a RSA1 key file /home/waswc/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/waswc/.ssh/id_rsa type 1
debug1: identity file /home/waswc/.ssh/id_rsa-cert type -1
debug1: identity file /home/waswc/.ssh/id_dsa type -1
debug1: identity file /home/waswc/.ssh/id_dsa-cert type -1
debug1: identity file /home/waswc/.ssh/id_ecdsa type -1
debug1: identity file /home/waswc/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version conker_5f1b1e4ced 1dd7a6fe6754
debug1: no match: conker_5f1b1e4ced 1dd7a6fe6754
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 768 bytes for a total of 789
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96
debug2: kex_parse_kexinit: hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-s
debug2: kex_parse_kexinit: ssh-ed25519,ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-256,rsa-sha2-512
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug2: dh_gen_key: priv key bits set: 160/320
debug2: bits set: 1010/2048
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug3: Wrote 272 bytes for a total of 1061
debug3: check_host_in_hostfile: host bitbucket.org filename /home/waswc/.ssh/known_hosts
debug3: check_host_in_hostfile: host bitbucket.org filename /home/waswc/.ssh/known_hosts
debug3: key_read: type mismatch
debug3: check_host_in_hostfile: match line 139
debug3: check_host_in_hostfile: host 104.192.141.1 filename /home/waswc/.ssh/known_hosts
debug3: check_host_in_hostfile: host 104.192.141.1 filename /home/waswc/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 135
debug1: Host 'bitbucket.org' is known and matches the ECDSA host key.
debug1: Found key in /home/waswc/.ssh/known_hosts:139
debug2: bits set: 1012/2048
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1077
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 52 bytes for a total of 1129
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/waswc/.ssh/identity ((nil))
debug2: key: /home/waswc/.ssh/id_rsa (0x55ef37201f80)
debug2: key: /home/waswc/.ssh/id_dsa ((nil))
debug2: key: /home/waswc/.ssh/id_ecdsa ((nil))
debug3: Wrote 68 bytes for a total of 1197
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/waswc/.ssh/identity
debug3: no such identity: /home/waswc/.ssh/identity
debug1: Offering public key: /home/waswc/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 372 bytes for a total of 1569
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug2: input_userauth_pk_ok: SHA1 fp 9d:32:fe:7c:d3:38:5f:b2:85:b0:a9:f8:71:b8:57:bf:73:52:63:25
debug3: sign_and_send_pubkey: RSA 9d:32:fe:7c:d3:38:5f:b2:85:b0:a9:f8:71:b8:57:bf:73:52:63:25
debug1: read PEM private key done: type RSA
debug3: Wrote 644 bytes for a total of 2213
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug3: Wrote 68 bytes for a total of 2281
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: callback start
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env DB2INSTANCE
debug3: Ignored env IBM_DB_LIB
debug3: Ignored env rvm_bin_path
debug3: Ignored env HOSTNAME
debug3: Ignored env GEM_HOME
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env HISTSIZE
debug3: Ignored env IRBRC
debug3: Ignored env SSH_CLIENT
debug3: Ignored env IBM_DB_DIR
debug3: Ignored env QTDIR
debug3: Ignored env MY_RUBY_HOME
debug3: Ignored env QTINC
debug3: Ignored env SSH_TTY
debug3: Ignored env USER
debug3: Ignored env LD_LIBRARY_PATH
debug3: Ignored env LS_COLORS
debug3: Ignored env _system_type
debug3: Ignored env rvm_path
debug3: Ignored env IBM_DB_HOME
debug3: Ignored env rvm_prefix
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env MODULEPATH
debug3: Ignored env LOADEDMODULES
debug3: Ignored env _system_arch
debug3: Ignored env _system_version
debug3: Ignored env DB2_HOME
debug3: Ignored env SSH_ASKPASS
debug3: Ignored env HISTCONTROL
debug3: Ignored env rvm_version
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env IBM_DB_INCLUDE
debug3: Ignored env LOGNAME
debug3: Ignored env QTLIB
debug3: Ignored env CVS_RSH
debug3: Ignored env CLASSPATH
debug3: Ignored env GEM_PATH
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env MODULESHOME
debug3: Ignored env LESSOPEN
debug3: Ignored env DB2LIB
debug3: Ignored env RUBY_VERSION
debug3: Ignored env _system_name
debug3: Ignored env G_BROKEN_FILENAMES
debug3: Ignored env BASH_FUNC_module()
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: Wrote 120 bytes for a total of 2401
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
authenticated via ssh key.

You can use git to connect to Bitbucket. Shell access is disabled
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: channel 0: close_fds r -1 w -1 e 6
debug3: Wrote 36 bytes for a total of 2437
debug3: Wrote 68 bytes for a total of 2505
Transferred: sent 2304, received 2368 bytes, in 0.3 seconds
Bytes per second: sent 8574.5, received 8812.6
debug1: Exit status 0

--------------------------------------------------------------------
FAILURE:
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/waswc/.ssh/config
debug3: key names ok: [ecdsa-sha2-nistp256,ssh-rsa,ssh-dss]
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to bitbucket.org [104.192.141.1] port 22.
debug1: Connection established.
debug1: identity file /home/waswc/.ssh/identity type -1
debug1: identity file /home/waswc/.ssh/identity-cert type -1
debug3: Not a RSA1 key file /home/waswc/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/waswc/.ssh/id_rsa type 1
debug1: identity file /home/waswc/.ssh/id_rsa-cert type -1
debug1: identity file /home/waswc/.ssh/id_dsa type -1
debug1: identity file /home/waswc/.ssh/id_dsa-cert type -1
debug1: identity file /home/waswc/.ssh/id_ecdsa type -1
debug1: identity file /home/waswc/.ssh/id_ecdsa-cert type -1
ssh_exchange_identification: Connection closed by remote host

In the meanwhile, I am going to open also a ticket.

Thank you very much in advance,

Alberto

[Theodora Boudale]

Hi Alberto,

Thank you for the info. I don't see anything in the output that would indicate why this occurs. Please create the support ticket and provide this info, it can then be escalated to our engineering team for further investigation.

Kind regards,
Theodora