How do I set up ssh public-key authentication so that I can use ssh, sftp or scp from my Bitbucket Pipelines pipeline?

[Philip Hodder · Accepted solution]

04/04/2018 update

We've updated the SSH functionality in Bitbucket Pipelines, and I would suggest you following the documentation here: https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html

The responses here are still valid, but are much more complicated to set up than the above documentation.

[Ganesh Kunwar]

This did not work for me. Is there anything that I can miss?

[Philip Hodder]

Can you open a new question with some specific error messages you're getting. Then send me a link to it here. Thanks.

[Steven Vaccarella]

Firstly, here's an overview of what you need to accomplish:

1. Generate a public/private key pair.

2. Store the private key in a secure Pipelines environment variable.

3. Upload the public key to the server that you want to communicate with from your pipeline.

4. Add a "known_hosts" file to your repository that includes the server's public ssh key.

5. Add some commands to the beginning of your pipeline script to tie everything together.

And now the details:

1. Generate a public/private key pair.

   ssh-keygen -t rsa -N '' -f my_ssh_key

   This will create two files - a private key named "my_ssh_key" (without a password) and a public key named "my_ssh_key.pub".

2. Store the private key in a secure Pipelines environment variable. We'll base-64 encode it to make sure it survives the trip through the Pipelines UI.

   base64 < my_ssh_key

   Just copy and paste the output of this command into a Pipelines environment variable (be sure to tick the "secure" checkbox). I'll assumed you've named the variable "MY_SSH_KEY".
   
   IMPORTANT: Please be aware that ticking the "secure" checkbox will not prevent someone who has push access to your repository (and thus can control what happens when a pipeline is executed) from retrieving the key.
    

3. Upload the public key to the server that you want to communicate with from your pipeline. There are various ways to do this and if you're using a fully managed service you may be able to upload the key via a web page or similar UI. However if you have ssh access to the server then the simplest method is probably using the ssh-copy-id tool.

   ssh-copy-id -i my_ssh_key.pub username@server.example.com

   This uploads your new public key to the right location so that the ssh server will trust it for accessing the specified user's account. Typically that means appending the key to the ~/.ssh/authorized_keys file on the server.
   
   At this point it may be worth verifying that you can ssh into the server using the new key without having to enter a password (assuming that you expect to have full ssh access to the server).

   ssh -i my_ssh_key username@server.example.com

4. Add a "known_hosts" file to your repository that includes the server's public ssh key. One way to do this is using the "ssh-keyscan" utility:

   ssh-keyscan -t rsa server.example.com > my_known_hosts

   Commit the my_known_hosts file to your repository so that it can be accessed from your pipeline. If you don't have the ssh-keyscan utility then another method is to just copy an existing known_hosts file from the ~/.ssh directory of a user that has previously accessed the server via ssh. It is a simple text file containing one line per server - you should be able to copy the file and delete all the lines except the single line for the server you want to connect to.

5. Add the following commands to the beginning of your pipeline script to tie everything together.

   - mkdir -p ~/.ssh
   - cat my_known_hosts >> ~/.ssh/known_hosts
   - (umask 077; echo $MY_SSH_KEY | base64 --decode > ~/.ssh/id_rsa)

Now you should be able to run ssh-based commands without passwords. Here's an example of uploading a file using sftp:

- sftp username@server.example.com <<< $'put file-to-upload.txt'

[ckirksey]

Would be nice to edit with @Antonello Moro's comment

[Kashif-Ur-Rehman]

@Steven Vaccarella, I followed you above tutorial step by step, and I was successful in getting file (my war file). 
But when i try to get that file on my local setup (through FTP), which is windows, it takes too much time i.e 3 hours.
 

Is there any way that we can transfer our file from pipeline to windows server instead of linux ? 

[Steven Vaccarella]

I'm not sure I understand your question. In your current setup are you uploading the file from your pipeline to an external Linux server and then downloading the file from that server to your local Windows machine? If that's the case then your download time is going to be affected primarily by your internet connection speed. I'm not sure how uploading the file to a Windows server instead is going to help.

At any rate it should be possible to upload to a Windows machine by following a similar process if it is running an ssh-based file server. You'll need to consult the documentation of the file server to work out how to do step 3 (configure the file server to trust your ssh key) and step 4 (obtain the server's public ssh key to add to your known hosts file).

[Kashif-Ur-Rehman]

@Steven Vaccarella, Thanks, Nicely explained.

[Kashif-Ur-Rehman]

@Steven Vaccarella,

I am using windows.
I have followed above partial steps. Partial steps means, I cannot do step 3 and 4. When I try to run these two step i did not get any error but I cannot get key and known host file .
But here is my issue.
I have accesss one of the remote server, there i setted up FTP.
I also setted up key base authentication to that FTP.  With key base authentication, I can login with putty/winscp to my FTP server.
Now I added base64 version of my private key to environment variable of pipeline, and then also committed my_known_host file, which only contains public key contents. Nut I am not able to send war to my FTP.

Will you please help me.

[Matt T]

@Steven Vaccarella

Sorry, the guide is a little confusing, where should I be creating the keys - on the remote server, or on bitbucket cloud via pipelines?

Thanks

[King]

Either. You can create it on your remote server (lets say, an EC2 instance) and add them back to your BB repo. Or the other way, so your EC2 instance knows and trusts the key you created in BB.

[Philip Hodder]

This is still a valid answer to the question. But I've unaccepted this to highlight the new SSH key setup flows.

[Ramesh Depaiah]

@Steven Vaccarella  When i check from git bash it says logged in success but when i try it from my pipelibe with same public key it fails I've uploaded the SSH key to bitbucket 

[Antonello Moro]

In case you should run into this error:

base64: invalid input

Then just add -i at your base64 decoding command, like this:

- (umask 077; echo $MY_SSH_KEY | base64 --decode -i > ~/.ssh/id_rsa)

[Ranjeet Ranjan]

I follow your tutorial but still it asking password. 

[Steven Vaccarella]

Did you mean to comment on this answer or on my answer above? Are you sure that you used an empty password when you created the keypair? (the -N '' argument to the ssh-keygen command).

If you post your pipeline's script and the exact log output we might be able to help further.

[Matt MacGillivray]

in my testing, the "echo $MY_SSH_KEY" statement always returns "$MY_SSH_KEY" which fits with the Bitbucket pipelines documentation - secured variables can't be echoed... so the example doesn't work?

[Steven Vaccarella]

The echo command works normally. It's your logs that get altered so you can't see secure variables.

[Kashif-Ur-Rehman]

?

[Kashif-Ur-Rehman]

@Steven Vaccarella, I followed your steps, but at end of step 3 i.e 
ssh-copy-id -i my_ssh_key.pub username@server.example.com

It is still asking for password ?

[Steven Vaccarella]

Note that at step 3 the ssh-copy-id command is just one possible method of uploading your public key to the server and it may or may not be applicable depending on the type of server and what sort of access you already have to that server. The example command I provided will work if you have ordinary ssh access to the server using password-based authentication. If you don't have password-based access but you have key-based access (ie. using a different key from the one you just generated) then it may still be possible to use the ssh-copy-id command but you may have to supply different arguments - check the command's docs for more information. Or you may find it easier to simply ssh into the server and manually append the new public key to the "authorized_keys" file in the appropriate user account. Or (again, depending on the server) you may have other options available. For example if the server is Bitbucket itself then you can upload the public key using the web interface. 

[Kashif-Ur-Rehman]

@Steven Vaccarella, I am able to proceed with my previous error.
But now I am not able to get Artifat (my war file ).

Here is the script that I wrote on bitbucket-pipelines.yml

image: maven:3.3.9
  
pipelines:
  default:
    - step:
        script:  # Modify the commands below to build and test your repository.
          - mkdir -p ~/.ssh
          - cat known_hosts >> ~/.ssh/known_hosts
          - (umask 077; echo echo $MY_SSH_KEY | base64 --decode -i > ~/.ssh/id_rsa)
          - cd EmailService
          - mvn clean package
          - sftp root@203.215.163.16 <<< $'put EmailService-1.0-SNAPSHOT.war'

Basically when I want is , I want to send war file (which will be created when mvn clean package commands run successfully) to my ftp server.

But on command  

sftp root@203.215.163.16 <<< $'put EmailService-1.0-SNAPSHOT.war'

It is taking infinite time.

My purpose if to get war file, at moment what i know is there is no way to get war file from bitbucket pipeline it can only send war to FTP server or any other cloud environment.
 

[Steven Vaccarella]

Are you uploading from the correct directory? Normally maven creates artifacts in the "target" subdirectory, but it looks like you're trying to upload from the root project directory ("EmailService").

[Kashif-Ur-Rehman]

Okay, So it means to execute this statement 

 scp EmailService-1.0-SNAPSHOT.war root@203.215.163.16

first I have to cd to Target folder , then transfer file
i.e

cd target

scp EmailService-1.0-SNAPSHOT.war root@203.215.163.16

right ?

One more confusion is that logs of command:-

mvn clean package
says that build is successfull
and war is here:-
Building war: /opt/atlassian/pipelines/agent/build/EmailService/target/EmailService-1.0-SNAPSHOT.war

means whether I have to execute this statement :-
 

scp ~/opt/atlassian/pipelines/agent/build/EmailService/target/EmailService-1.0-SNAPSHOT.war

root@203.215.163.16


or  

cd target

scp EmailService-1.0-SNAPSHOT.war root@203.215.163.16


?

[Steven Vaccarella]

The directory "/opt/atlassian/pipelines/agent/build/" is the initial working directory where your code is checked out and where your script starts executing. You already have a command "cd EmailService" in your script (before running the maven command), so doing another "cd target" from there should get you into the right directory. Give it a try 

[Kashif-Ur-Rehman]

Okay,So I tried it, and it worked fine for me. I am able to transfer war file to my FTP server.
Thanks. 

[Kashif-Ur-Rehman]

@Steven Vaccarella, Now I am trying to write script in gradle to get WAR file
here is .yml file script:-

image: qlik/gradle
 
pipelines:
  default:
    - step:
        script:
          - mkdir -p ~/.ssh
          - cat my_known_hosts >> ~/.ssh/known_hosts
          - (umask 077; echo $MY_SSH_KEY | base64 --decode -i > ~/.ssh/id_rsa)
          - gradle --version
          - cd EmailService
          - gradle test
          - gradle war
          - sftp root@203.215.163.16 <<< $'put EmailService-1.0-SNAPSHOT.war'
          

All commands successfully completed. Build is successful.

But at last command i.e   

sftp root@203.215.163.16 <<< $'put EmailService-1.0-SNAPSHOT.war'

Logs says:-

Connected to 203.215.163.16.
sftp> put EmailService-1.0-SNAPSHOT.war
stat EmailService-1.0-SNAPSHOT.war: No such file or directory


and if I change like

- cd target
- sftp root@203.215.163.16 <<< $'put EmailService-1.0-SNAPSHOT.war'

then logs says that :-

bash: cd: target: No such file or directory

Can you tell me, what is going wrong here ? 
I am not sure whether I used correct gradle command to create war file.
and where war file placed when gradle is used (like for maven it placed under target folder) 

Connected to 203.215.163.16.

Connected to 203.215.163.16.

[Steven Vaccarella]

I'm not sure where gradle builds its artifacts. I'd recommend either consulting the gradle documentation or running the build locally to find out 

[Kashif-Ur-Rehman]

@Steven Vaccarella, One thing i noticed that for maven based project setup.
I got the War file, But war size is not same as that war which I created manually.
Do i have to change anything in build script in  bitbucket-pipelines.yml i.e 
 

image: maven:3.3.9
  
pipelines:
  default:
    - step:
        script:  # Modify the commands below to build and test your repository.
          - mkdir -p ~/.ssh
          - cat known_hosts >> ~/.ssh/known_hosts
          - (umask 077; echo echo $MY_SSH_KEY | base64 --decode -i > ~/.ssh/id_rsa)
          - cd EmailService

                    -  mvn clean package
                    - cd target
                    - sftp root@203.215.163.16 <<< $'put EmailService-1.0-SNAPSHOT.war'

[Steven Vaccarella]

We seem to have strayed a bit off topic for this question. It would be better if you ask this in a new question - that way more people will see it so you'll be more likely to get an answer and other people will be more likely to benefit from the answers.

[rskn]

Hello all ,

After I have done all of the steps , at the end I am still being asked for passphrase.

Here is my log : 

mkdir -p ~/.ssh 

+ mkdir -p ~/.sshCompleted at 2017-01-29T16:04:58Z

 cat my_known_hosts >> ~/.ssh/known_hosts 

+ cat my_known_hosts >> ~/.ssh/known_hostsCompleted at 2017-01-29T16:04:58Z

 (umask 077; echo $SSH_PRIVATE_KEY | base64 --decode -i> ~/.ssh/id_rsa) 

+ (umask 077; echo $SSH_PRIVATE_KEY | base64 --decode -i> ~/.ssh/id_rsa)Completed at 2017-01-29T16:04:58Z+ sftp 

$SERVER_USER@$SERVER_ADDRESSEnter passphrase for key '/$SERVER_USER/.ssh/id_rsa': 

I need to transfer .jar file after mvn clean package to my remote server (running on ubuntu server)

[Steven Vaccarella]

It looks like you created your key pair using a passphrase. You should create the key pair without a passphrase for this process to work (the -N '' option in step 1).

[swhite]

Antonello, thank you. Your single comment to this community saved me hours of grief. (I'd already had days of it.)

To others: if you still receive an error such as "SSH public key authentication failed: Unable to extract public key from private key file" or "Permission denied (publickey)" adapt the above answer to include the following:

- (umask 077; echo $SSH_PUBLIC_KEY > ~/.ssh/id_rsa.pub)

This is necessary because Bitbucket still doesn't pass the public key even though we input them in Settings -> SSH Keys. I could never get this to work (for weeks) until I added the above.

Then, you can do something like this:

- git ftp init --key ~/.ssh/id_rsa --pubkey ~/.ssh/id_rsa.pub --user username sftp://hostname/path/

What a nightmare, Bitbucket.

[Alesclandre]

Hello, I followed your tutorial exactly as you said but it still asks for passwrod, although  I created the key as you said (ssh-keygen -t rsa -N '' -f my_ssh_key): 

ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory

The only thing I changed is the last line (which gives me the error) : 

ssh -T $USER_DEV@$IP_ADDRESS -p $PORT

in order to be directly on the server and make a git clone from there.

Didn't try sftp though...

But this doesn't work. Am I doing something wrong ?

Even bitbucket suggests things that does not work...

[EVO_9]

Just typed the following into terminal and I get: command not found. Using Terminal and MacOS if that makes a difference

base64 < my_ssh_key

[Kashif-Ur-Rehman]

I am using windows.
I have followed above partial steps. Partial steps means, I cannot do step 3 and 4. When I try to run these two step i did not get any error but I cannot get key and known host file .
But here is my issue.
I have accesss one of the remote server, there i setted up FTP.
I also setted up key base authentication to that FTP.  With key base authentication, I can login with putty/winscp to my FTP server.
Now I added base64 version of my private key to environment variable of pipeline, and then also committed my_known_host file, which only contains public key contents. Nut I am not able to send war to my FTP.

Will you please help me.

[Kashif-Ur-Rehman]

How can i use java_home from pipeline gradle script.
Actually I need rt.jar for compiling my one project, I cannot commit rt.jar because its size is very big, so I was just finding any other possible way of how I can compile my project without commiting rt.jar.
Is there any way that i can refer rt.jar from java_home of pipeline ?

[Kashif-Ur-Rehman]

@Steven Vaccarella , I am following same steps mentioned by you, but here I am transferring War file to Windows server instead of Linux.

For generating public/private keys I am using puttyGen software and then I saved private key to Pipeline Environment variable. But while this step:-
sftp xxx@205.216.163.91 <<< $'put xxx.war'

it throws an error of
"Host key verification failed.
Couldn't read packet: Connection reset by peer."

Can you please help me out ?

One more thing, I was not able to do step no 3 through command line i.e
ssh-copy-id -i my_ssh_key.pub xxx@205.216.163.91

Actually When I run this command It do some processing then throws an error of
"Umask" command not recognized.

For SSH/SFTP server setting on windows, I was following this URL :-
https://winscp.net/eng/docs/guide_windows_openssh_server#installing_sftp_ssh_server

[edrdesigner]

Use sftp -o StrictHostKeyChecking=no -o port="22" -o IdentityFile="~/.ssh/id_rsa"