Private self hosted npm repository package on bitbucket using pipelines

Question

I'm trying to use a self hosted (in bitbucket) npm package. I was able to use it locally without any issues, but when running the bitbucket pipeline I get an error. It fails on the npm install step.

This is what I have on the repo using the private npm package:

...
"dependencies": {
"shared-lib": "ssh://git@bitbucket.org/{org-name}/shared-lib.git"
...
}

Both repositories are under the same organization (private) in bitbucket.

On the library that is consuming the self hosted private npm package I created a key by going to: Settings > Pipelines > SSH Keys (self generated key on bitbucket)

On the shared-lib, I have added this SSH key by going to Settings > SSH Keys > Add key (self generated).

Also added bitbucket.org as known address.

The bitbucket-pipelines.yml at this moment does not have any special configuration, i'm only running npm install (where it fails) and npm run test.

This is the configuration I have, but still fails when running the pipeline.

This is the error when running the pipeline:

npm ERR! syscall spawn git
npm ERR! path git
npm ERR! errno ENOENT
npm ERR! enoent Error while executing:
npm ERR! enoent undefined ls-remote -h -t ssh://git@bitbucket.org/{org-name}/shared-lib.git
npm ERR! enoent 
npm ERR! enoent 
npm ERR! enoent spawn git ENOENT
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent```

Magdziarz_ Slawomir · Answer

For anyone coming across this answer from google search results (future myself included): my bitbucket pipeline was failing in the step when I ran "npm ci"; git was returning a "Permission denied (publickey)" error, even though I had followed the instructions from above and copied the ssh key from the MY_SSH_KEY variable (repository variable in my case) into ~/.ssh/id_ed25519.

When debugging, it turned out that git was actually using another ssh key (/opt/atlassian/pipelines/agent/ssh/id_rsa) which was defined in repository settings (in the "Pipelines > SSH Keys" section).

So I ended up using that key instead, and copied its public part to the other repository which was the npm dependency.

Theodora Boudale · Answer

Hi Sarah,

If I understand correctly, you are running Bitbucket Pipelines builds in 80 different repositories, and each of them needs to clone 30 private repositories?

If so, what you could do instead is

* generate an SSH key pair locally
* use a workspace variable named e.g. MY_SSH_KEY where you store the private SSH key after you base-64 encode it

This variable can then be accessed by all 80 repos, assuming they all belong to the same workspace.

The public key can be added to Projects > Projects settings > Access keys of the project or projects to which the 30 repos belong.

You will still need to edit the bitbucket-pipelines.yml file of all 80 repos to create a ~/.ssh/known_hosts file and also create the private key from the variable MY_SSH_KEY.

There are more details about this solution here:

* https://support.atlassian.com/bitbucket-cloud/docs/use-multiple-ssh-keys-in-your-pipeline/

The benefit of this approach is that you have one SSH key and if you ever need to change it in the future, you change it in only one place (the workspace variable).

Please keep in mind that there are security risks associated with this approach, I am copying from the documentation:

There are security risks associated with passing private SSH keys as repository variables:

* Repository variables get copied to child processes that your pipelines build may spawn.
* Secured variables can be retrieved by all users with write access to a repository.

We recommend that you never pass your own personal SSH key as an repository variable, but instead generate a new SSH key-pair for Pipelines that easily be disabled if it is compromised. It may also be worth using deployment variables, which you can combine with deployment permissions to control access.

The documentation is referring to repository variables, but these apply to workspace variables as well.

Since a workspace variable can be retrieved by all users with write access to any of these 80 repos, this means that any of these users could retrieve the variable and clone any of the 30 repos that the public key gives access to.

Additionally, adding an SSH key to a project's Access keys will grant access to all repositories of that project.

You will need to weigh the risks and benefits to decide which approach would be best for you.

Kind regards,
Theodora

sarah_veysset · Answer

I have the same issue as Luis and git is well installed. The problem is that I have multiple private repositories in my dependencies (around thirty) and I have around 80 projects that use all these dependencies. It's impossible for me to set up so many keys (80 x 30). Is there another solution?

Kind regards,
Sarah