How to get a pipeline to fetch (or be sent) a file(s) from a remote server

Adam Brunt_s7f5w1

Hi all,

I am well versed with using the atlassian/sftp-deploy pipe for sending files to a remote server but - unless I am missing something obvious - that only PUTs files it can't be used to GET them. Again, I might be forgetting something obvious, but is there anyway I can get a file(s) sent to the pipeline - ideally to the artefacts folder for the current run ?

Basically, in one step we SSH onto a remote server and generate some files (on that server) which we then want to be available to a subsequent step in the pipeline. The files would NOT be available publicly via cURL or an FTP server. Is this possible ? Maybe I am just missing some knowledge about what an SSH connection can/can't do i.e can you fetch a file over an SSH connection ?

Theodora Boudale

Hi Adam,

This should be possible with scp. For example:

scp $USER@$SERVER:/path/to/somefile.txt $BITBUCKET_CLONE_DIR/artifacts

where /path/to/somefile.txt is the path and the name of the file on the remote server. The file will then be copied in a directory named artifacts inside the clone directory.

In order to use an SCP command, an SSH client needs to be installed in the Docker image you use as a build container (for the step where you want to fetch files).

You can check that locally if you have Docker installed on your computer. For example, if you use atlassian/default-image:4 as a build container for this step, you run locally

docker run -it --entrypoint=/bin/bash atlassian/default-image:4

Then, when you are inside the container, you run

ssh -V

If an SSH client is installed, you will see something like the following:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022

If an SSH client is not installed, you will need to either install it during the build or use a different Docker image with an SSH client preinstalled.

Since you SSH into this server, I assume that SSH keys are already set up. If you use SSH keys configured in Repository settings > SSH keys, you can also adjust the command as follows to ensure this specific SSH key is offered:

scp -i /opt/atlassian/pipelines/agent/ssh/id_rsa $USER@$SERVER:/path/to/somefile.txt $BITBUCKET_CLONE_DIR/artifacts

If you are not using the SSH keys configured in Repository settings but a variable that stores the base64-encoded value of the SSH key, you will need to create the private key in this step as well and then you can pass its path and name with the -i argument to the scp command.

Kind regards,
Theodora