How can I use SSH in Bitbucket Pipelines?

[Philip Hodder]

I would like to use SSH in Bitbucket Pipelines to clone another repository from Bitbucket.

[Philip Hodder · Accepted solution]

Up to date as of 04/04/2018.

For example. I want to clone repo-2 into a build from repo-1

To use SSH to clone another repository from Bitbucket you need to do the following.

1. In repo-1 generate a Pipelines SSH key to be injected into your build container. Follow Step 1 here: https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html
   • Note that if the known hosts information for bitbucket.org and github.com are injected by default. If you are using any other site you will need to configure Known Hosts, by following Step 2 in the above documentation.
2. Add an Access Key to repo-2. Paste in the public key that was generated in the previous step.
3. Configure your bitbucket-pipelines-yml to use git.

An example bitbucket-pipelines.yml is:

image: ubuntu:16.04

pipelines:
  default:
    - step:
        - apt-get update -y
        - apt-get install -y git
        - git clone <your_repo_here> # For example, git@bitbucket.org:username/repo-2.git

[Khue Bui]

Hi Philip,

I followed your steps above to add submodule from another bitbucket repo to my pipeline, but unfortunately encounter error:

 Host key verification failed.fatal: Could not read from remote repository.

[GC]

Hi @Philip Hodder! I work at my client's office and they have provided me access to their private repositories. All repositories that i have access to are private. I am trying to run a pipeline in repo A and want to checkout some build scripts from repo B so i could use them during the pipeline run in repo A. I followed your steps described above and added public key (from the key-pair i generated in repo A) to "Access keys" section in repo B. Now when i try building code, i get the following message in pipeline for repo A:

+ git clone -b dev git@bitbucket.org:company_name/devops.git
Cloning into 'devops'...
Warning: Permanently added the RSA host key for IP address '18.205.93.0' to the list of known hosts.

To access this repository, add your IP address to the whitelist. 
For details, see https://confluence.atlassian.com/x/TY5qMw

fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

I checked the link mentioned in the log but it seems whitelisting is a Premium feature. When i checked settings in my avatar, it shows free account. Is there no way i can access one private repo's content in another repo's pipeline without a Premium account?

[Philip Hodder]

Hi @GC,

The IP whitelist needs to be configured on your client's account (the one that owns the repository you're trying to clone). Not on your own user account.

The IPs that need to be whitelisted are listed here: https://confluence.atlassian.com/bitbucket/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall-343343385.html

Specifically the "Valid IP addresses for Bitbucket Pipelines build environments" section.

Thanks,

Phil

[GC]

Thanks for the response @Philip Hodder. I'll request the client for whitelisting however, the IP mentioned in my error message is 18.205.93.0 but the section you've pointed out isn't having that range. On the other hand, under section "Valid IP addresses for bitbucket.org, api.bitbucket.org, and altssh.bitbucket.org", i see
"(Since July 28, 2018) IPv4 inbound for bitbucket.org, api.bitbucket.org, and altssh.bitbucket.org" which has my range. Can you kindly confirm which range do i need to whitelist?

[Philip Hodder]

I think you still need to use the IP range I suggested.

The IP mentioned in your build is being added to the known_hosts inside your pipeline. i.e. Your pipelines is talking to Bitbucket to perform the clone (IP 18.205.93.0 in this case)

If the range I suggest doesn't work, then try the other ones and report back. :)

[GC]

That makes sense. Thanks for clarifying it. I've sent both the range sets to the client. Thanks Philip! :)

[Philip Hodder]

No problem :)

[André Ferro Cadete]

@GC

If both repos reside in the same owner (i.e. the team) you don't need much to check-out a Repo within another Repo's pipeline. All you need is to configure the SSH key pairs within Pipeline itself, this will give you a public key that you can add to the authorized keys of the other Repo and it is enough.

Creating the SSH key specifically for Pipelines is described here:

https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html

The last part, instead of adding the public key to a "remote server" you will add it in the authorized keys of the repository you want to clone in the pipeline of the other repository.

(i.e. Repo A creates Pipeline key, Repo B adds as authorized key. Now Repo A can clone repo B in it's pipelines without explicitly pointing to a key).

Edit: answering because it does not sound like a whitelist problem, maybe something else you missed? You can also add the fingerprint of bitbucket in the same place you create the pipelines ssh key, but it shouldn't be needed (bitbucket already accepts its own host).

[GC]

@André Ferro CadeteThanks for the response but when you said adding public key to authorized keys file in repo B (that i want to clone inside repo A's pipeline), where exactly will i do that? In case of a running pipeline, it's possible but where can this be done in a repo which needs to be cloned?

[GC]

@Philip Hodder: It's working now after whitelisting the IPs. Thank you! :)

[Alejandro Cotilla]

This solution works for me if I make git operations in my yml file, but, it doesn't work if my Docker image makes git operations from within. How can solve that? Thanks!

This is the error I get:

Please make sure you have the correct access rights
and the repository exists.
Warning: Permanently added the RSA host key for IP address '(...)' to the list of known hosts.
repository access denied. deployment key is not associated with the requested repository.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

[Alejandro Cotilla]

Never mind, the issue was because the SSH keys defined under Settings / Pipelines / SSH keys have read-only access and I was trying to push.

[Samuel Dorsaz · Accepted solution]

There is today an official way to use keys without env variables

https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html

[Elliot Borst]

How do you actually use that key in a command?

[Philip Hodder]

The keys are injected into your build and should be available to commands that need them (in a similar way to how it would run locally).

Note that you'll still need to add the corresponding public key to whichever service you're trying to contact.

[Elliot Borst]

Yeah ive dropped the pipeline pub key onto my server in .ssh/authorized_keys

A simple "ssh -i user@domain.com" doesn't work

Ill keep trying

[Philip Hodder]

What error message are you getting? Key not found, or authentication errors?

[Samuel Dorsaz]

@Elliot Borstthe -i option let you change the default key but you don't specify any file. You should remove this option.

[Elliot Borst]

"ssh root@server.domain.com  Pseudo-terminal will not be allocated because stdin is not a terminal.Host key verification failed."  

The pub key is on the other box, I can ssh fine from my Mac, I can see the connection come through our firewall ok.  Seems like pipelines isn't sending the private key with my command

[Elliot Borst]

Oh I worked it out, I wasn't actually adding the known host after doing the lookup.

[Samuel Dorsaz]

Pseudo-terminal will not be allocated because stdin is not a terminal

take a look on ssh -t option if necessary

[Hassange]

> Oh I worked it out, I wasn't actually adding the known host after doing the lookup.

Same issue, thanks for the tip

[Steven Vaccarella]

For a slightly more general guide for using ssh and ssh-based tools from a pipeline (not just for cloning another Bitbucket repository) see this other question:

https://answers.atlassian.com/questions/39429257

It is similar to Philip's answer but also covers server key fingerprints and uploading your public key to other types of servers.

[Philip Hodder]

Out of date response. Content here still works, but is much more work to set up.

See this for an up to date solution: https://community.atlassian.com/t5/Bitbucket-questions/Re-How-can-I-use-SSH-in-Bitbucket-Pipelines/qaq-p/764824/comment-id/25670#M25670

--------------------------

Old method

There's a few steps required to do this.

First, you need to generate a key-pair to use for Bitbucket.

Generate a new SSH key-pair for use in Bitbucket. 

$ ssh-keygen -t rsa -b 4096 -C "

Do not use a passphrase, when prompted.

Once you have generated the key-pair, add the public key to your account:

https://confluence.atlassian.com/display/BITBUCKET/Add+an+SSH+key+to+an+account

If you already had an SSH key, and skipped the set up, start reading from here.

Now, we can set up SSH in Bitbucket Pipelines:

First, we will add the SSH key as an environment variable.

As of right now, Pipelines does not support line breaks in the environment variable, so we need to encode the private key first:

$ base64 <path>/<to>/id_rsa | pbcopy # probably path is ~/.ssh/id_rsa

Now create a secured environment variable in Bitbucket Pipelines called PRIVATE_KEY, with the contents of the base64 encoded private key:

https://confluence.atlassian.com/display/BITBUCKET/Environment+variables+in+Bitbucket+Pipelines

Now, create a bitbucket-pipelines.yml with the following content (I can't find the original source of this example sorry, share if you know ):

# You can use a Docker image from Docker Hub or your own container
# registry for your build environment.
pipelines:
  default:
    - step:
        script: # Modify the commands below to build your repository.
          - echo $PRIVATE_KEY > ~/.ssh/id_rsa.tmp
          - base64 -d ~/.ssh/id_rsa.tmp > ~/.ssh/id_rsa
          - chmod 600 ~/.ssh/id_rsa
          - base64 ~/.ssh/id_rsa
          - git clone git@bitbucket.org:<account_name>/<repo_name>.git

You should see that the build successfully cloned your repository.

Troubleshooting

Q: The clone asks for a passphrase:

A: There are two things you should double check. When you created the SSH key-pair, did you create them without a passphrase? (If there's a passphrase, things get much more complicated than expected). If you did not have a passphrase, double check that the key (the private key) you have stored is base64 encoded.

------------------------

Out of date response. Content here still works, but is much more work to set up.

See: https://community.atlassian.com/t5/Bitbucket-questions/Re-How-can-I-use-SSH-in-Bitbucket-Pipelines/qaq-p/764824/comment-id/25670#M25670

[Rob Jones]

 This line is superfluous and can be deleted: 
- base64 ~/.ssh/id_rsa

 

[Keith Webster]

I added the above to my config file but on the first statement, echo $PRIVATE_KEY > ~/.ssh/id_rsa.tmp, I received the following: bash: /root/.ssh/id_rsa.tmp: No such file or directory

[Philip Hodder]

Add this statement before you echo the PRIVATE_KEY:

mkdir ~/.ssh

Does that help? 

[Pintea Cosmin]

Maybe this will help... I have configured the  SSH key-pair without passphrase but when i tried to clone the repo with the above code i get the the request to enter some passphrase . For me the solution was to add -i to the command that write the key   . I dont know why i have this behavior. The encoding was done with version  8.25 and decoded with 8.21.

base64 -d  -i ~/.ssh/id_rsa.tmp > ~/.ssh/id_rsa

[Riyanka Purkayastha]

after running
git clone git@bitbucket.org:account_name/repo_name.git

+ git clone git@bitbucket.org:account_name/repo_name.git
Cloning into 'repo_name'...
The authenticity of host 'bitbucket.org (104.192.143.3)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?

and can't do anything else

[Pintea Cosmin]

try this : 

 - ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts

[Steven Vaccarella]

That will work, but it's not very secure (it's essentially bypassing host key checking because the key scan is performed on every build). A better option is to do the key scan just once locally and then include the full host key in your script or add a known_hosts file to your repository. That way your build will fail if the host key changes (which would indicate a potential security issue). You can find more details in step 4 of the following answer: How do I set up ssh public-key authentication so that I can use ssh, sftp or scp from my Bitbucket Pipelines pipeline?

[André Ferro Cadete]

As of right now, Pipelines does not support line breaks in the environment variable 

@Steven Vaccarella
Would you PLEASE add this potentially time-saving information to your ENV var documentation page?

https://confluence.atlassian.com/bitbucket/environment-variables-794502608.html

This info is crucial and nowhere to be found in the docs. Wasted a few hours trying to understand what the hell was wrong with my keys. 

For Docker within Docker + composer install, your new "easy" solution does not work (my docker build will run composer install and inside the built container your Bitbucket keys are nowhere to be found, nor do I want to rely on it - my build tool should work anywhere, not specifically in bitbucket pipelines).

Thanks

[Linette]

@André Ferro CadeteGood call! I've updated the environment variables page accordingly. Sorry you had such a painful experience, but hopefully now others won't have the same.

Linette
(Pipelines writer)

[André Ferro Cadete]

Hi @Linette,

Thanks for the quick response and action. And no problem, was a learning experience as always ;)

[Nguyễn Hồng Quân]

Please support SSH key of ed25519 type.

[Philip Hodder]

Hi,

Can you please open a feature request for that here: https://bitbucket.org/site/master/issues/new

[Cory Robinson]

I'm attempting to use `npm` packages installed from private repositories in the same team account as the pipelines repo is running. I tried the steps for adding ssh but no luck any advice?

[anton_bakai_twinwin]

I have the same issue when I'm trying to install a dependency from a private repo. Did you find the solution?

[Cory Robinson]

@Philip Hodder any ideas on how we could solve this? 

[Philip Hodder]

Can you list out the steps you've tried for setting this up at the moment? It'll make it easier for me to see if you've missed anything in particular.

From the error message, it looks like NPM isn't using SSH. It's calling an HTTPS URL, and mentions an invalid username and password, rather than an invalid SSH key.

[Raja Periyasamy]

@Cory Robinson Hope you resolved this issue. I had the very same issue and resolved by:

- changed the dependency git url in the package file from https to ssh. i.e. https://bitbucket.org/<company>/<project>.git to git@bitbucket.org:<company>/<project>.git.

- added the pipeline ssh public key to the target project.

[Keith Webster]

Great, this worked. Thanks!