setup a piple line with sftp copy file

[Mohamed Bakr]

I have a pipe line with 2 steps , i have create ssh and added public key to my server

- step:
    name: deployment
    runs-on:
     - 'self.hosted'
    script:
    - pipe: atlassian/ssh-run:0.4.0
      variables:
       SSH_USER: 'ubuntu'
       SERVER: 172.0.1.195
       MODE: 'command'
       DEBUG: 'true'
       COMMAND: 'echo ${BITBUCKET_COMMIT::7} > a.txt'
    - pipe: atlassian/sftp-deploy:0.5.11
      variables:
        USER: 'ubuntu'
        SERVER: '172.0.1.195'
        REMOTE_PATH: '/home/ubuntu'
        DEBUG: 'true'

but when i tried to execute this pipeline ssh pass and sftp or scp always fail with the following logs

Runner matching labels:
- linux
- self.hosted
Runner name: ta3meed-sandbox-runner
Runner labels: self.hosted, linux
Runner version:
current: 1.333
latest: 1.336

The version of this runner is outdated. Upgrade to the latest version (1.336). 
+ umask 000

+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="deployment-pipeline" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR
Cloning into '/opt/atlassian/pipelines/agent/build'...

+ git reset --hard bc6100786f45d22fc8ef283281d49687c8e08409
HEAD is now at bc61007 add debug

+ git config user.name bitbucket-pipelines

+ git config user.email commits-noreply@bitbucket.org

+ git config push.default current

+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/

+ git remote set-url origin http://bitbucket.org/$BITBUCKET_REPO_FULL_NAME

+ git reflog expire --expire=all --all

+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude

+ chmod 777 $BUILD_DIR

Images used:
build: atlassian/default-image@sha256:3a09dfec7e36fe99e3910714c5646be6302ccbca204d38539a07f0c2cb5902d4
docker: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon@sha256:03bfa3875b810fe280239175f742a20c500d9b1f181c83a3b05cdc04b1cfb12c

+ docker container run \
--volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \
--volume=/usr/local/bin/docker:/usr/local/bin/docker:ro \
--volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/ssh-run:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/ssh-run \
--workdir=$(pwd) \
--label=org.bitbucket.pipelines.system=true \
--env=BITBUCKET_STEP_TRIGGERER_UUID="$BITBUCKET_STEP_TRIGGERER_UUID" \
--env=BITBUCKET_REPO_FULL_NAME="$BITBUCKET_REPO_FULL_NAME" \
--env=BITBUCKET_GIT_HTTP_ORIGIN="$BITBUCKET_GIT_HTTP_ORIGIN" \
--env=BITBUCKET_PROJECT_UUID="$BITBUCKET_PROJECT_UUID" \
--env=BITBUCKET_REPO_IS_PRIVATE="$BITBUCKET_REPO_IS_PRIVATE" \
--env=BITBUCKET_WORKSPACE="$BITBUCKET_WORKSPACE" \
--env=BITBUCKET_REPO_OWNER_UUID="$BITBUCKET_REPO_OWNER_UUID" \
--env=BITBUCKET_BRANCH="$BITBUCKET_BRANCH" \
--env=BITBUCKET_REPO_UUID="$BITBUCKET_REPO_UUID" \
--env=BITBUCKET_PROJECT_KEY="$BITBUCKET_PROJECT_KEY" \
--env=BITBUCKET_REPO_SLUG="$BITBUCKET_REPO_SLUG" \
--env=CI="$CI" \
--env=BITBUCKET_REPO_OWNER="$BITBUCKET_REPO_OWNER" \
--env=BITBUCKET_STEP_RUN_NUMBER="$BITBUCKET_STEP_RUN_NUMBER" \
--env=BITBUCKET_BUILD_NUMBER="$BITBUCKET_BUILD_NUMBER" \
--env=BITBUCKET_GIT_SSH_ORIGIN="$BITBUCKET_GIT_SSH_ORIGIN" \
--env=BITBUCKET_PIPELINE_UUID="$BITBUCKET_PIPELINE_UUID" \
--env=BITBUCKET_COMMIT="$BITBUCKET_COMMIT" \
--env=PIPELINES_JWT_TOKEN="$PIPELINES_JWT_TOKEN" \
--env=BITBUCKET_STEP_UUID="$BITBUCKET_STEP_UUID" \
--env=BITBUCKET_CLONE_DIR="$BITBUCKET_CLONE_DIR" \
--env=BITBUCKET_DOCKER_HOST_INTERNAL="$BITBUCKET_DOCKER_HOST_INTERNAL" \
--env=DOCKER_HOST="tcp://host.docker.internal:2375" \
--env=BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" \
--env=BITBUCKET_PIPE_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/ssh-run" \
--env=COMMAND="echo ${BITBUCKET_COMMIT::7} > a.txt" \
--env=DEBUG="true" \
--env=MODE="command" \
--env=SERVER="172.0.1.195" \
--env=SSH_USER="ubuntu" \
--add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL" \
bitbucketpipelines/ssh-run:0.4.0
Unable to find image 'bitbucketpipelines/ssh-run:0.4.0' locally
0.4.0: Pulling from bitbucketpipelines/ssh-run
31603596830f: Pulling fs layer
c507f8e37a23: Pulling fs layer
7eca4d8c4c7b: Pulling fs layer
c21a713484c2: Pulling fs layer
c21a713484c2: Waiting
7eca4d8c4c7b: Verifying Checksum
7eca4d8c4c7b: Download complete
31603596830f: Download complete
c507f8e37a23: Verifying Checksum
c507f8e37a23: Download complete
31603596830f: Pull complete
c21a713484c2: Verifying Checksum
c21a713484c2: Download complete
c507f8e37a23: Pull complete
7eca4d8c4c7b: Pull complete
c21a713484c2: Pull complete
Digest: sha256:2aa2f6517fb364da07a18f8b1e55b651c1db0361f7a850b79914c8da663f709e
Status: Downloaded newer image for bitbucketpipelines/ssh-run:0.4.0
INFO: Executing the pipe...
INFO: Using default ssh key
INFO: Executing command on 172.0.1.195
ssh -A -tt -i /root/.ssh/pipelines_id -o StrictHostKeyChecking=no -p 22 ubuntu@172.0.1.195 bash -c 'echo bc61007 > a.txt'
Warning: Permanently added '172.0.1.195' (ECDSA) to the list of known hosts.
Connection to 172.0.1.195 closed.
✔ Execution finished.

+ docker container run \
--volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \
--volume=/usr/local/bin/docker:/usr/local/bin/docker:ro \
--volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/sftp-deploy:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/sftp-deploy \
--workdir=$(pwd) \
--label=org.bitbucket.pipelines.system=true \
--env=BITBUCKET_STEP_TRIGGERER_UUID="$BITBUCKET_STEP_TRIGGERER_UUID" \
--env=BITBUCKET_REPO_FULL_NAME="$BITBUCKET_REPO_FULL_NAME" \
--env=BITBUCKET_GIT_HTTP_ORIGIN="$BITBUCKET_GIT_HTTP_ORIGIN" \
--env=BITBUCKET_PROJECT_UUID="$BITBUCKET_PROJECT_UUID" \
--env=BITBUCKET_REPO_IS_PRIVATE="$BITBUCKET_REPO_IS_PRIVATE" \
--env=BITBUCKET_WORKSPACE="$BITBUCKET_WORKSPACE" \
--env=BITBUCKET_REPO_OWNER_UUID="$BITBUCKET_REPO_OWNER_UUID" \
--env=BITBUCKET_BRANCH="$BITBUCKET_BRANCH" \
--env=BITBUCKET_REPO_UUID="$BITBUCKET_REPO_UUID" \
--env=BITBUCKET_PROJECT_KEY="$BITBUCKET_PROJECT_KEY" \
--env=BITBUCKET_REPO_SLUG="$BITBUCKET_REPO_SLUG" \
--env=CI="$CI" \
--env=BITBUCKET_REPO_OWNER="$BITBUCKET_REPO_OWNER" \
--env=BITBUCKET_STEP_RUN_NUMBER="$BITBUCKET_STEP_RUN_NUMBER" \
--env=BITBUCKET_BUILD_NUMBER="$BITBUCKET_BUILD_NUMBER" \
--env=BITBUCKET_GIT_SSH_ORIGIN="$BITBUCKET_GIT_SSH_ORIGIN" \
--env=BITBUCKET_PIPELINE_UUID="$BITBUCKET_PIPELINE_UUID" \
--env=BITBUCKET_COMMIT="$BITBUCKET_COMMIT" \
--env=PIPELINES_JWT_TOKEN="$PIPELINES_JWT_TOKEN" \
--env=BITBUCKET_STEP_UUID="$BITBUCKET_STEP_UUID" \
--env=BITBUCKET_CLONE_DIR="$BITBUCKET_CLONE_DIR" \
--env=BITBUCKET_DOCKER_HOST_INTERNAL="$BITBUCKET_DOCKER_HOST_INTERNAL" \
--env=DOCKER_HOST="tcp://host.docker.internal:2375" \
--env=BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" \
--env=BITBUCKET_PIPE_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/sftp-deploy" \
--env=DEBUG="true" \
--env=REMOTE_PATH="/home/ubuntu" \
--env=SERVER="172.0.1.195" \
--env=USER="ubuntu" \
--add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL" \
bitbucketpipelines/sftp-deploy:0.5.11
Unable to find image 'bitbucketpipelines/sftp-deploy:0.5.11' locally
0.5.11: Pulling from bitbucketpipelines/sftp-deploy
31603596830f: Already exists
9e006a152763: Pulling fs layer
8c449c3c01f1: Pulling fs layer
1bd77f40acd7: Pulling fs layer
8c449c3c01f1: Verifying Checksum
8c449c3c01f1: Download complete
1bd77f40acd7: Verifying Checksum
1bd77f40acd7: Download complete
9e006a152763: Verifying Checksum
9e006a152763: Download complete
9e006a152763: Pull complete
8c449c3c01f1: Pull complete
1bd77f40acd7: Pull complete
Digest: sha256:9fcd9a8ec73daf0ec51461079b1853fb6b5f93970842f6d80e64160f4b5535ff
Status: Downloaded newer image for bitbucketpipelines/sftp-deploy:0.5.11
+ SFTP_DEBUG_ARGS=-v
+ [[ -z '' ]]
+ info 'Using SSH.'
INFO: Enabling debug mode.
INFO: Using SSH.
+ echo -e '\e[36mINFO: Using SSH.\e[0m'
+ setup_ssh_dir
+ INJECTED_SSH_CONFIG_DIR=/opt/atlassian/pipelines/agent/ssh
+ IDENTITY_FILE=/opt/atlassian/pipelines/agent/ssh/id_rsa_tmp
+ KNOWN_HOSTS_FILE=/opt/atlassian/pipelines/agent/ssh/known_hosts
+ mkdir -p /root/.ssh
+ touch /root/.ssh/authorized_keys
+ [[ -z '' ]]
+ '[' '' '!=' '' ']'
+ '[' '!' -f /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp ']'
+ debug 'Using default ssh key'
+ [[ true == \t\r\u\e ]]
+ echo -e '\e[37mDEBUG: Using default ssh key\e[0m'
+ cp /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp /root/.ssh/pipelines_id
DEBUG: Using default ssh key
+ '[' '!' -f /opt/atlassian/pipelines/agent/ssh/known_hosts ']'
+ cat /opt/atlassian/pipelines/agent/ssh/known_hosts
+ '[' -f /root/.ssh/config ']'
+ [[ -z '' ]]
+ echo 'IdentityFile ~/.ssh/pipelines_id'
+ chmod -R go-rwx /root/.ssh/
+ run_pipe
INFO: Starting SFTP deployment to 172.0.1.195:/home/ubuntu...
DEBUG: Executing echo "mput /opt/atlassian/pipelines/agent/build/*" | sftp -b - -rp -v ubuntu@172.0.1.195:/home/ubuntu
+ info 'Starting SFTP deployment to 172.0.1.195:/home/ubuntu...'
+ echo -e '\e[36mINFO: Starting SFTP deployment to 172.0.1.195:/home/ubuntu...\e[0m'
+ set +e
+ [[ -z '' ]]
+ debug Executing echo '"mput' '/opt/atlassian/pipelines/agent/build/*"' '|' sftp -b - -rp -v ubuntu@172.0.1.195:/home/ubuntu
+ [[ true == \t\r\u\e ]]
+ echo -e '\e[37mDEBUG: Executing echo "mput /opt/atlassian/pipelines/agent/build/*" | sftp -b - -rp -v ubuntu@172.0.1.195:/home/ubuntu\e[0m'
+ echo 'mput /opt/atlassian/pipelines/agent/build/*'
+ sftp -b - -rp -v ubuntu@172.0.1.195:/home/ubuntu
OpenSSH_7.9p1, OpenSSL 1.1.1g 21 Apr 2020
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 172.0.1.195 [172.0.1.195] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/pipelines_id type -1
debug1: identity file /root/.ssh/pipelines_id-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3
debug1: match: OpenSSH_8.9p1 Ubuntu-3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.0.1.195:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:cBkkJ0ADEjHdkR6jqj+XaNobKRAyefvZ4jp/Ub3mPtA
Host key verification failed.

Connection closed.
Connection closed
✖ Deployment failed.
+ STATUS=255
+ set -e
+ [[ 255 == \0 ]]
+ fail 'Deployment failed.'
+ echo -e '\e[31m✖ Deployment failed.\e[0m'
+ exit 255
Searching for files matching artifact pattern .bitbucket/pipelines/generated/pipeline/pipes/**

Searching for test report files in directories named [test-reports, TestResults, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.

[Theodora Boudale · Accepted solution]

Hi @Mohamed Bakr and welcome to the community!

If you open the repository with this pipeline on Bitbucket website, go to its Repsoitory settings > SSH keys, is the address of your server in the Known hosts section?

The error "Host key verification failed" indicates that it may not be there.

If it is not there, can you add it and select Fetch to fetch the host's fingerprint?
After it is successfully added, please run this pipeline again and let me know how it goes.

If you still see errors, please copy-paste the new log so we can further investigate.

Kind regards,
Theodora

[Mohamed Bakr]

Thanks Theodora for your update , but I am running this pipeline on self-host agent and for local network we are establishing the communication to our application server that we need to copy files their,

as you see IP address is 

172.0.1.195

which is reside on the same network of our agent , so it's not applicable to add it to know hosts list. 

[Theodora Boudale]

Hi @Mohamed Bakr,

Builds that use Linux runners run in a Docker container on your machine, they do not run directly on your machine.

The SSH keys and known_hosts file that you have on your machine with the runner are not accessible and cannot be used by the Docker container that runs your build.

• Can you please confirm how you have set up SSH?
  Have you generated a new key pair from Repository settings > SSH key?
  Have you added your own key pair on that page?
  Or do you have an SSH key on the Docker image you are using?

The ~/.ssh/known_hosts file inside the Docker container (not the one on your machine) needs to be updated with the public key of the server you want to send files to. The easiest way to do this is by adding the address of the server on the Known hosts section in the Repository settings of this repo > SSH keys.

Alternatively, you can also try passing an argument to the pipe and set StrictHostKeyChecking to no, to prevent this error without updating known_hosts. You could do this as follows:

- pipe: atlassian/sftp-deploy:0.5.11
  variables:
    USER: 'ubuntu'
    SERVER: '172.0.1.195'
    REMOTE_PATH: '/home/ubuntu'
    DEBUG: 'true'
    EXTRA_ARGS: '-o StrictHostKeyChecking=no'

Kind regards,
Theodora

[Mohamed Bakr]

Thanks @Theodora Boudale  Extra Args solved my problem 

[Igor Stoyanov]

@Mohamed Bakr  hi. Thanks for your question.

Since you are using self hosted runner, you should try to pass your ssh key directly to the pipe:

script:
  - pipe: atlassian/ssh-run:0.4.1
    variables:
      SSH_USER: 'ec2-user'
      SERVER: '127.0.0.1'
      SSH_KEY: $MY_SSH_KEY
      MODE: 'script'
      COMMAND: 'myscript.sh' # path to a script in your repository

SSH_KEY is a base64 encoded alternate SSH_KEY to use instead of the key configured in the Bitbucket Pipelines admin screens (which is used by default). This should be encoded.

To encode ssh key:

Linux

 $ base64 -w 0 < my_ssh_key

Mac OS X

 $ base64 < my_ssh_key

Best regards, Igor.

[Mohamed Bakr]

@Igor Stoyanov  

ssh Pipe is working fine with me , but ftp pipe has a problem , and @Theodora Boudale  helped me to resolve this issue using extra args to all unkhown host names