Pipelines rsync-deploy with multiple SSH keys - known_hosts issue

[rossberenson]

I'm having difficulty getting multiple ssh keys to work with rsync-deploy. I was able to get this working with one key, imported into Bitbucket but when I tried to configure it to work with multiple, that's where I ran into challenges.

I followed Use multiple SSH keys in your pipeline to the T. Multiple times. I keep running into the issue where the known_hosts file is not being found. I've tried connecting with the snippet example provide and I receive the error below.

ssh $USER@$DEV_SERVER -p $SSH_PORT 'echo "connected to `host` as $USER"'
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Host key verification failed.

I've also tried using rsync-deploy and run into the same issue. I enabled debug and receive these errors:

debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug2: resolving $DEV_SERVER port $SSH_PORT
debug3: resolve_host: lookup $DEV_SERVER:$SSH_PORT
debug3: ssh_connect_direct: entering
debug1: Connecting to 
 [70.32.23.79] port $SSH_PORT.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /root/.ssh/pipelines_id type -1
debug1: identity file /root/.ssh/pipelines_id-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: compat_banner: match: OpenSSH_7.4 pat OpenSSH_7.4* compat 0x04000006
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to $DEV_SERVER:$SSH_PORT as 'rossbere'
debug3: put_host_port: [$DEV_SERVER]:$SSH_PORT
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: no algorithms matched; accept original
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:uNLTiehjO9mZF2U0e4/zdKwj/jtBouEX7wicp1j9/cA
debug3: put_host_port: [70.32.23.79]:$SSH_PORT
debug3: put_host_port: [$DEV_SERVER]:$SSH_PORT
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: checking without port identifier
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: hostkeys_find_by_key_hostfile: trying user hostfile "/root/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: hostkeys_find_by_key_hostfile: trying user hostfile "/root/.ssh/known_hosts2"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /root/.ssh/known_hosts2 does not exist
debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts does not exist
debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts2"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts2 does not exist
debug1: read_passphrase: can't open /dev/tty: No such device or address
Host key verification failed.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]

Some more information. This is for a WordPress website. .git is located in the root. In the root is also wp-content, bitbucket-pipelines.yml, and my_known_hosts. My YML below. It's a bit messy, but trying to get this work, then will clean it up. 

image: atlassian/default-image:3

pipelines:
  branches:
    dev:
      - step:
          name: 'Deployment to Dev'
          deployment: dev
          script:
            - mkdir -p ~/.ssh
            - cat my_known_hosts >> ~/.ssh/known_hosts
            # - (umask  077 ; echo $DEV_SSH_KEY | base64 --decode > ~/.ssh/id_rsa)
            # - ssh $DEV_USER@$DEV_SERVER -p $DEV_PORT 'echo "connected to `host` as $DEV_USER"'
            # - ls
            # - nvm install v18.16.0
            # - nvm use v18.16.0
            # - cd wp-content/themes/bfi # Change to the directory containing your package.json
            # - npm install
            # - npm run build
            - pipe: atlassian/rsync-deploy:0.12.0
              variables:
                USER: $DEV_USER
                SERVER: $DEV_SERVER
                REMOTE_PATH: $REMOTE_PATH
                LOCAL_PATH: build
                DEBUG: 'true'
                SSH_KEY: $DEV_SSH_KEY
                SSH_PORT: $DEV_PORT
                SSH_ARGS: '-vvv'
                EXTRA_ARGS: '--archive --delete --compress --progress --partial --verbose --exclude=node_modules --exclude=src --exclude=.git'
                DEBUG: 'true' # Optional.
                # DELETE_FLAG: '' # Optional.

Any help would be appreciated.

[Theodora Boudale · Accepted solution]

Hi @rossberenson and welcome to the community!

There are two different errors here. The first one is:

read_passphrase: can't open /dev/tty: No such device or address

and the second one is:

Host key verification failed.

The first error indicates that Pipelines cannot read the SSH key's passphrase (this is not possible to do in Pipelines, since you cannot interact with the build and provide a passphrase while the build runs). Please generate an SSH key pair without a passphrase and use that in Pipelines instead.

Regarding the second error message, instead of using a my_known_hosts file in your repo, you can go to Repository settings (on Bitbucket website) > section PIPELINES - SSH Keys. At the bottom of the page, there is a section Known hosts. You can enter the domain name or the IP address of the server(s) you want to connect to (please include the port if it's different than the standard port 22 for SSH) and fetch the fingerprint. If you do that, there is no need to keep a my_known_hosts file in your repo or create this file during the build.

Please feel free to let me know how it goes and if you need further assistance.

Kind regards,
Theodora

[rossberenson]

Thanks for getting back to me. I resolved the issue. I started over. I realized I had conflicting data for known_hosts. I removed my file, and redid the known hots built into bitbucket. 

Thank you!

[Theodora Boudale]

You are very welcome and thank you for the update. Please feel free to reach out if you ever need anything else!