I have a repo where I want to do the deploy over the rsync-deploy pipe in my bitbucket-pipeline
however it keeps failing on SSH with the message: Permission denied (publickey).
to simplify testing I have added the following line to the start of my deploy script which it keeps failing on as well.
- ssh -v $SSH_USER@$SSH_SERVER_LIVE "ls -l /srv"
I'll tell you a bit more about how i have set things up.
The repo has a public/private keypair that is being reused over a number of other repos and servers which all have no issues.
the public key of this repo is known to the server I am trying to connect to. It is included in the authorized_keys file located in ~/.ssh
I have added the IP to the known hosts and generated a fingerprint for it. This IP is what is set to the $SSH_SERVER_LIVE
besides with the IP i also tried it with the Public DNS provided by AWS as wel as a custom DNS we use internally. They all generate the same fingerprint and they all fail when sshing from within the pipeline.
This setup is identical to other repos which have no SSH issues, I also set up a fresh test repo with the same config that only includes the pipeline yml which only executes the command above and it works fine.
I will now provide some logging, cleared of sensitive information.
failing SSH step in the offending repo:
+ ssh -v $SSH_USER@$SSH_SERVER_LIVE "ls -l /srv"
OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.9 30 May 2023
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to {ip-address-removed} [{ip-address-removed}] port 22.
debug1: Connection established.
debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa type -1
debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.3
debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to {ip-address-removed}:22 as 'ubuntu'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:{fingerprint-known_hosts-removed}
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '{ip-address-removed}' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:11
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /opt/atlassian/pipelines/agent/ssh/id_rsa explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /opt/atlassian/pipelines/agent/ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
ubuntu@{ip-address-removed}: Permission denied (publickey).
successful SSH step in the test repo:
+ ssh -v $SSH_USER@$SSH_SERVER "ls -l /srv"
OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.9 30 May 2023
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to {ip-address-removed} [{ip-address-removed}] port 22.
debug1: Connection established.
debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa type 0
debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.3
debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to {ip-address-removed}:22 as 'ubuntu'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256: {fingerprint-known_hosts-removed}
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '{ip-address-removed}' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:7
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /opt/atlassian/pipelines/agent/ssh/id_rsa RSA SHA256:{fingerprint-not-in-known_hosts-removed} explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /opt/atlassian/pipelines/agent/ssh/id_rsa RSA SHA256:{fingerprint-not-in-known_hosts-removed} explicit
debug1: Server accepts key: /opt/atlassian/pipelines/agent/ssh/id_rsa RSA SHA256:{fingerprint-not-in-known_hosts-removed} explicit
Authenticated to {ip-address-removed} ([{ip-address-removed}]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for {ip-address-removed} / (none)
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for {ip-address-removed} / (none)
debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: Sending command: ls -l /srv
Learned new hostkey: RSA SHA256:{fingerprint-not-in-known_hosts-2-removed}
Learned new hostkey: ED25519 SHA256:{fingerprint-not-in-known_hosts-3-removed}
Adding new key for {ip-address-removed} to /root/.ssh/known_hosts: ssh-rsa SHA256:{fingerprint-not-in-known_hosts-2-removed}
Adding new key for {ip-address-removed} to /root/.ssh/known_hosts: ssh-ed25519 SHA256:{fingerprint-not-in-known_hosts-3-removed}
debug1: update_known_hosts: known hosts file /root/.ssh/known_hosts2 does not exist
debug1: pledge: fork
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
total 8
drwxrwxrwx 12 root root 4096 Aug 30 14:49 maple-server
drwxr-xr-x 2 root root 4096 Aug 30 14:28 mongo-gateway
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 5408, received 5044 bytes, in 0.8 seconds
Bytes per second: sent 6792.6, received 6335.4
debug1: Exit status 0
/var/log/auth.log server for offending repo, failed
sshd[30921]: AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:{fingerprint-not-in-known_hosts-removed} failed, status 22
sshd[30921]: Connection closed by authenticating user ubuntu {ip-address-removed} port 46835 [preauth]/var/log/auth.log server for test repo, success
sshd[30871]: Accepted publickey for ubuntu from {ip-address-removed} port 40194 ssh2: RSA SHA256:{fingerprint-not-in-known_hosts-2-removed}
sshd[30871]: pam_unix(sshd:session): session opened for user ubuntu(uid=1000) by (uid=0)
systemd-logind[376]: New session 249 of user ubuntu.
sshd[30917]: Received disconnect from {ip-address-removed} port 40194:11: disconnected by user
sshd[30917]: Disconnected from user ubuntu {ip-address-removed} port 40194
sshd[30871]: pam_unix(sshd:session): session closed for user ubuntu
systemd-logind[376]: Session 249 logged out. Waiting for processes to exit.
systemd-logind[376]: Removed session 249.fingerprints removed with a placeholder like {fingerprint-not-in-known_hosts-removed} are fingerprints that are not the same as found in settings/ssh-keys on the repo
I honestly have no idea what is going wrong or what is left for me to try and change. Any input is appreciated. Hope I have ben clear about the issue. I will try to clarify if needed.
Thanks in advance.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.