Request for Comment (RFC) - Step-level Variable Sharing

[#1]

Hi all - this is the first RFC we've done via this community, but we'd like to make this a regular thing depending on how engagement goes with this attempt.

This RFC is to gather ideas/feedback on our implementation for Step-level Variable Sharing. A new feature coming soon to Pipelines.

Summary

This feature will allow users to share environment variables between steps. A variable can be defined in one step, and subsequent steps in the same pipeline will have access to that variable.

Motivation

Currently, there is no standard way to achieve this. There are workarounds available, the one most often used is to save variables in an artifact file, and use that artifact in subsequent steps to set variables. This is a functional solution, but not recommended because

• It involves custom logic that is not using artifacts for their intended purpose

• It’s annoying and non-intuitive for customers to use this approach

• Variables produced this way can easily override other variables when they maybe should not

Example Usage

Users can save variables to a step-level variable object. At completion of the step, these variables are extracted and saved as pipeline-level variables, which are imported into subsequent steps.

step:
  script:
    echo "MY_VARIABLE=some_value" >> $BITBUCKET_STEP_VARIABLES

Key Points:

• Variables produced by a step are saved as generic pipeline-level variables. They will be injected into all subsequent steps using the variable name defined when they were created initially.
• If multiple steps write to the same variable name, the system will follow a "last write wins" strategy.
• Variables will be captured by the system at the end of each step, not at the point when the variable is written in the script.
• Variables will be injected into steps when the step starts and will not be updated (in the context of the running step) if the value of the variable (at the pipeline level) changes during a steps execution.
• This avoids instability due to race-conditions between steps causing variable values to change mid-step.
• The current state of all pipeline-level variables will be printed to the step setup logs when each step starts. This will provide users with an immutable record of the state of all variables for each specific step.

Key Questions:

• What level of priority should be given to step-level variables if they are written with the same name as Workspace/Repo/Environment/Custom variables?
  • e.g. Should variables captured from a step take highest priority, leaving pre-set variables to act more like "fallback values"?

[Peter Falck Grony]

Hi Edmund,

I would prefer that the step-level variables take the highest priority. As you mention, then the pre-set values can be utilized as fallback values which would fit very well with the practices of my organization.

[Jason Harrison]

Step level variables should be applied as late as possible in the initialization of the step.

My concern is the printing of the pipeline-level variables in the step setup logs, specifically how will secrets be obscured? 

[Edmund Munday]

Hi @Jason Harrison - to be VERY clear, this feature is NOT designed to be a secure/encrypted mechanism to pass secrets between steps.

We strongly recommend utilising a dedicated 3rd party secret management tool via our generic 3rd Party Secret provider interface.

Hypothetically, if you wanted to pass things like one-time-use secrets around between steps, you would need to encrypt/decrypt them yourself - maybe storing the keys as secured variables.

This is not something we would recommend though as the values of those secrets would NOT be masked in the logs if they were ever accidentally printed.

Given the risk/liability aspects of this kind of thing, support for secure/encrypted value sharing is explicitly not in scope for this feature. This should be seen as a metadata/state-management capability, not a secret sharing capability.

[Jason Harrison]

@Edmund Munday I understand. My question is "if there is an environment variable holding a secret, how do we prevent it from being passed to the following steps, and how do we prevent it's value from being included in the step setup logs?

[Edmund Munday]

Hey @Jason Harrison - so I think I understand your question now, sorry.

Just to confirm, the risk you're raising is someone using this feature to effectively "proxy" (for lack of a better term) the value of a secret out of the build context it's made available in via an exported variable?

So for example:

// Existing secret MY_SECURED_VARIABLE exists in the environment.

step:
  script:
    echo "MY_NON_SECURED_VARIABLE=$MY_SECURED_VARIABLE" >> $BITBUCKET_STEP_VARIABLES

step:
  script:
    echo $MY_NON_SECURED_VARIABLE // use this to exfiltrate the secret from the previous step.

I'll raise this with the team and make sure we implement safeguards to prevent this kind of scenario. It's already something we have to address with masking variables in the build logs.

Thank you for raising this, it's greatly appreciated.

[Jason Harrison]

@Edmund Munday 

from your description of the feature:

• The current state of all pipeline-level variables will be printed to the step setup logs when each step starts. This will provide users with an immutable record of the state of all variables for each specific step.

This is where I interpreted a pipeline-level secret being output without masking.

[PierrickPuimeanChieze]

@Jason HarrisonIf I understand correctly, you are not really concerned about the scenario outlined by @Edmund Munday in his post, but more that this new feature could print pipeline-defined secret in the output ?

[Jason Harrison]

@PierrickPuimeanChieze correct