Is information sharing between pipelines possible?

[Dewald van Hoven]

Hi Atlassian Community

My team and myself have a use case where we want to share information between different pipelines while they are running. The successful execution of our integration tests is dependant on this information sharing.

We went quite far with how we wanted to implement the information sharing. The mechanism we thought would work is by sharing information between pipelines by using the BITBUCKET_PIPE_SHARED_STORAGE_DIR described here: https://support.atlassian.com/bitbucket-cloud/docs/advanced-techniques-for-writing-pipes/. We moved the integration tests to a custom pipe, described here: https://support.atlassian.com/bitbucket-cloud/docs/write-a-pipe-for-bitbucket-pipelines/, in order to gain access to the BITBUCKET_PIPE_SHARED_STORAGE_DIR.

We were naive as to how this machanism would work. We discovered that the information sharing can only happen between two pipes running within the same pipeline. Not pipes running in different pipelines.
Our current best way forward is to set up an external service where we can post to and read from to share information between pipelines. 

Is there a better way that we can do this from a mechanism in BitBucket?

Cheers

[Theodora Boudale · Accepted solution]

Hi @Dewald van Hoven,

What is possible is to share files that are produced by a step with a following step of the same pipeline, if you define certain files as artifacts:

• https://support.atlassian.com/bitbucket-cloud/docs/use-artifacts-in-steps/

If you do that for files generated in one step, these files will become available in the following steps.

Can you combine these different pipelines into one, with multiple steps? Or is that not possible (e.g. do you want to share info between a pipelines build running on develop branch and another one running on master)?

Kind regards,
Theodora

[Dewald van Hoven]

Hi @Theodora Boudale 

Thanks for your suggestion. Unfortunately it that will not work.

The key for us is to for completely different pipelines to be able to communicate with one another. 

We are building marketplace extensions for Confluence. In order to perform our integration tests, an app needs to register with a Confluence instance. Only one instance of an app can register to an instance at a time.

Out of a pool of Confluence instances, a pipeline should connect an instance another app is not connected to for a given run of the integration tests.

We can't find any mechanism to do this communication in the pipelines. The only way we can think of at the moment is to run a small external service which acts as a broker of instances a pipeline can connect to.

Kind regards

Dewald

[Theodora Boudale]

Hi Dewald,

Thank you for the info.

I'm afraid there is no way to have info shared between different pipelines builds.

If there is no way to get this info while a build is running and without communicating with other running pipelines:

a pipeline should connect an instance another app is not connected to for a given run of the integration tests

then the only way would be with a mechanism like the one you suggested.

Kind regards,
Theodora

[Dewald van Hoven]

Thanks Theodora :) 

[Theodora Boudale]

You are welcome Dewald, please feel free to reach out if you ever need anything else!