Access multiple Bitbucket repositories from a single Pipeline

[Michael Atkinson]

We currently have two repositories: one that contains the source code for our application, and another that contains scripts to build and deploy that application to the cloud (among other things). I want to set up two Pipelines: one in the application repo that simply triggers a Pipeline in the build repo upon commit to a specific branch, and another in the build repo that is triggered from the application repo and simply grabs the source from the application repo and executes the build process as normal. This is not a problem on our local machines, but it seems the only solution for Pipelines it to write a custom pipe that uploads the source to S3 (for example) in the application repo, and then pulls that same source down in the build repo. Is there anyway to use something like an app password to grant the build repo access to the application repo? Or is Pipelines just not the tool for this job?

[Theodora Boudale · Accepted solution]

Hi Michael,

It is possible to clone another repo in Pipelines. Instead of using app passwords, you can use SSH. The following blog post explains how you can set this up:

• https://bitbucket.org/blog/cloning-another-bitbucket-repository-in-bitbucket-pipelines

We also have a pipe that can trigger a pipeline for a different repo:

• https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/trigger-pipeline

You can use this in the yml file of the application repo in order to trigger a pipeline for the build repo. There is also a variable WAIT for the pipe that you can set to 'true' or 'false', depending on whether you want the pipeline in the application repo to wait for the pipeline in the build repo to complete or not.

I'm not sure if you're already aware of this pipe and if you are using it already, I thought I'd mention it just in case.

Please feel free to let me know if you have any questions.

Kind regards,
Theodora

[Michael Atkinson]

This is exactly what I was looking for, thank you Theodora!

[Theodora Boudale]

You are very welcome Michael, I'm glad to have helped!

[Sameeksha Chepe]

This is useful for the CI/CD I am setting up too. Thanks for this answer. 

However, I need to trigger a pipeline from another repo and use the build created by this pipeline. 

Will I be able to use 'artifacts' from the nested pipeline into the current pipeline which triggers the nested one? 

[Theodora Boudale]

Hi @Sameeksha Chepe,

Artifacts can only be used by subsequent steps of the same pipeline.

I'm afraid that it is not possible to transfer files from one Pipelines build to another one with the pipe.

One thing you could do would be upload these files to another storage server during the Pipelines build of the second/nested repo, and then when the pipe finishes, download them in the build that triggered the nested one.

Kind regards,
Theodora

[Sameeksha Chepe]

Thanks for the quick reply, @Theodora Boudale

Another variation of the question - I understand now that it's not possible using the pipes. 

However, we can save artifacts which are preserved for 14 days. Is it possible to use a command like cURL or something to copy the artifact from the client to the server repo during the pipeline run of of the server repo 

So the steps could be

* A pipeline runs on the client repo for a new commit on a specific branch. The build is stored as an artifact on the bitbucket artifacts storage.

* Whenever the pipeline for the server repo triggers, it copies the artifact from the latest build on client repo and creates a unified build of the server side and the client side code. 

[Theodora Boudale]

Hi @Sameeksha Chepe ,

I'm afraid that it is not possible to access artifacts via API.

Another way to work around this would be to upload the file to the Downloads section of the repo and then download it from there. My colleague Mark posted more details on the other question you have created here: https://community.atlassian.com/t5/Bitbucket-questions/How-to-use-build-artifact-from-one-pipeline-into-another/qaq-p/2025317

We have a feature request in our issue tracker for accessing artifacts via API: https://jira.atlassian.com/browse/BCLOUD-19330
It has been closed due to inactivity, as there was not a lot of demand for this feature. However, you can add your feedback and use case there if you'd be interested in this feature, as our product managers continue to monitor even closed requests.

Kind regards,
Theodora

[James Merkle]

If you require multiple repositories to be cloned, I would recommend disabling the automatic clone feature, and manually clone the repositories needed, especially if two repositories need to be "siblings", at the same directory level.

Add the following to the top of your YML file:

clone:
  enabled: false # do not perform a git clone into the build folder by default

Within your 'script' section, clone the repos necessary (power-shell environment):
'jack' is the "current repository", for which BITBUCKET_BRANCH environment will define the branch.

script:
 - env | sort
 - echo "Clone jack and jill repositories"
 - git clone -b $BITBUCKET_BRANCH --depth=1 git@bitbucket.org:my_id/jack.git # clone the branch requested
 - git clone --depth=1 git@bitbucket.org:my_id/jill.git

The result will be two repositories, 'jack', and 'jill', both in the build folder.
Using --depth=1 will limit history, allowing a faster clone process.