Multiple repos pushing to same repo - bitbucket pipeline fails

[Vinay Jindal]

I have a setup of multiple repos changing one central repo content by using their pipelines. The pipelines clone this central repo, change it and then push the change back to it. 

The issue that's happening is when first repo is changing the central repo, some other repo pushes the commit to it. This causes first repo's push to fail as the main branch is moved. 

Even if every repo takes a pull before pushing the change back, the same scenario can still happen. 

What's the best way to handle this type of setup using bitbucket pipelines ? Is there a way to lock the central repo or a way to automerge during git push ?

[Theodora Boudale · Accepted solution]

Hi @Vinay Jindal and welcome to the community!

What you are describing is standard behavior of Git.

One way I can think of to deal with this would be by writing a script that does the following:

• First, a git pull
• Then, a git push
• If git push returns a non-zero exit code, then try again a git pull and a git push.
• If git push returns a non-zero exit code again, repeat the previous step a number of times that is acceptable to you (e.g. in a for loop).

You could commit this script to each repo and execute it during the Pipelines build. Alternatively, you could upload it somewhere else (e.g. at the Downloads section of the repo) and download it during the build in order to run it.

Please keep in mind that this won't help in case there are conflicts when you pull.

However, if the push fails because the head of the branch is moved and there are no conflicts, then you need to try pulling and pushing again in order for it to succeed.

Kind regards,
Theodora

[Vinay Jindal]

thanks @Theodora Boudale for replying. This is the strategy that was in mind as well but I was looking for more cleaner solution to synchronise the repos. 

The 'Downloads' section is an interesting add it seems. Can you please share samples of how to download and upload a file to Downloads section through pipelines , we'll like to explore and integrate it as well. 

[Theodora Boudale]

Hi @Vinay Jindal,

Yes, I understand that this is not a very clean solution. It's the only one I can think of if you have builds that run on every push.

You could also use custom pipelines instead and then schedule them for each repo at a time frame different than the other repos:

• https://confluence.atlassian.com/bitbucket/scheduled-builds-for-pipelines-933078702.html

Then, the chances of the issue occurring are smaller (unless another user or script pushes also to the central repo). The disadvantage of this approach is that you won't have builds running as soon as a push occurs (I don't know if that is a requirement for you).

With regards to the Downloads section of the repo, here are some examples:

Uploading a file to Downloads

There are two ways:

1. You can use the following AI endpoint:

• https://developer.atlassian.com/cloud/bitbucket/rest/api-group-downloads/#api-repositories-workspace-repo-slug-downloads-post

curl --header "Authorization:Bearer ${repo_token}" -X POST https://api.bitbucket.org/2.0/repositories/${BITBUCKET_WORKSPACE}/${BITBUCKET_REPO_SLUG}/downloads -F files=@myScript.sh

where

repo_token is a user-defined repository variable holding the value of a Repository Access token
myScript.sh is the name of the script (the file should exist in the clone directory if you are executing this call during a Pipelines build)

This call uploads to the same repo where the build is running (BITBUCKET_WORKSPACE and BITBUCKET_REPO_SLUG) are default variables available to every build.

2. You could also use the following pipe to upload a file during a Pipelines build

• https://bitbucket.org/product/features/pipelines/integrations?search=downloads&p=atlassian/bitbucket-upload-file

Downloading a file from Downloads

You can use the following API endpoint:

• https://developer.atlassian.com/cloud/bitbucket/rest/api-group-downloads/#api-repositories-workspace-repo-slug-downloads-filename-get

curl --header "Authorization:Bearer ${repo_token}" -L https://api.bitbucket.org/2.0/repositories/${BITBUCKET_WORKSPACE}/${BITBUCKET_REPO_SLUG}/downloads/myScript.sh --output myScript.sh

If you use curl, it needs to be installed in the Docker image you have specified in your bitbucket-pipelines.yml file for the step. If it is not installed already, you can install curl during the build.

If you have any questions, please feel free to reach out!

Kind regards,
Theodora