Access private runner host environment variables

[clockworknet]

Hi,

We are utilising self hosted runners, with a runner located in each of our environments (dev, staging, prod, etc). Each of the runner hosts is built with a handful of global environment variables configured, that provide some general configuration for the environment in question (e.g. AWS_DEFAULT_REGION, ENVIRONMENT_NAME, etc).

Is there a way to pass these variables into the container used to run a step on those runners, the use case being that steps would inherit configuration, simply by virtue of the runner they are assigned to? (Similar to setting and using environment variables on individual Jenkins nodes) I would rather avoid having to duplicate this configuration inside BitBucket or having to maintain functionality inside the container to pull this info in from another source, if possible.

[Andrew Gibbs]

I would likewise love to be able to get an environment variable from the outer orchestrator container to the inner step container so I could make use of IAM Role derived credentials within my pipelines instead of having to use static credentials from an IAM User...

https://community.atlassian.com/t5/Bitbucket-questions/use-IAM-role-credentials-from-private-runner-executing-inside/qaq-p/1934915

[Theodora Boudale]

Hi @clockworknet and welcome to the community.

In order to provide isolation and reproducibility, a runner doesn’t provide access directly to the host file system, so I'm afraid that what you're asking is not possible.

You can either set these variables in Bitbucket or pass the values in a file and then source the file during the build.

Kind regards,
Theodora

[clockworknet]

Hi Theodora,

 I am suspecting that what I am asking about is not possible, but to be clear I am not asking for filesystem access. I am asking whether it is possible to pass environment variables that are set in the runner host globally, into the container started for each build step - the equivalent of:

`docker run -e SOME_GLOBAL_ENV_VAR ...`

[Theodora Boudale]

Hi @clockworknet,

Thanks for the clarification, I didn't realize that you were talking about passing the environment variables in the 'docker run' command for the runner.

Please allow me to check with my team internally and I'll get back to you on that.

Kind regards,
Theodora

[clockworknet]

Hi Theodora,

Thanks for looking into this further.

It is not the runner container that I am concerned about getting the environment variables into. I start that container and so have full control over the options that are used to start it.

It is each container that is started by the runner process, to handle a deployment step. I guess that if it were possible, those variables would need to be first passed into the runner container, but I am interested in whether it is possible to then get that process to pass them on to each container it starts.

[Norbert Csupka]

Hi @clockworknet 

Let me step in, I'm Norbert and I'm also a Bitbucket Cloud Support Engineer, it's nice to meet you.

I would like to inform you that the functionality what you would like achieve is unfortunately not possible. The only way how you can pass the environmental variables to the runner container is to set those variables in your Bitbucket Pipelines settings and once it's set up, those variables can be used in your build.

Please let me know if my explanation was clear and whether if you have any further question?

I'm looking forward to hear from you.

Have a great day ahead!

Best Regards,
Norbert
Atlassian Bitbucket Cloud Support Engineer

[Ajay _view26_]

HI @Norbert Csupka ,

Could you provide a sample example? 

I presume you are referring to using them as variables inside the pipeline.

Cheers

Ajay

[Norbert Csupka]

Hi Ajay,

Yes, of course please find the example :) .

I've created an "asdf" environmental variable with the value of "testtesttest" in Pipelines:

When I started this Pipelines, as you can see I'm using Runners and when I echo the "asdf" environmental variable, it shows the "testtesttest" value:

Please let me know whether if this is the functionality what you're looking for?

Best Regards,

Norbert 
Atlassian Bitbucket Cloud Support

[clockworknet]

Thanks @Norbert Csupkafor clarifying the situation - it would be a nice feature to have at some point if possible.

[Norbert Csupka]

Hi @clockworknet 

Can you clarify please what feature would you like to have implemented? Would you like to be able to pass the environmental variable via the docker run command? 

Once I have the clarification, I'm going to open a feature request for you.

I'm looking forward to hear from you.

Best Regards,
Norbert

Atlassian Bitbucket Cloud Support 

[clockworknet]

Hi @Norbert Csupka 

Our particular use case:

• we use existing automation and cfg mgmt to build our private runners hosts (in our case AWS EC2 instances), and deploy the BB Runner container instance (there is one small manual step in registering the runner in BB, where we need to copy the various tokens from BB)
• as part of this automated provisioning, we set a series of environment variables on the EC2 host. The values are typically derived from the AWS environment (e.g. tags added to the EC2 instance/param store values, etc) or otherwise stored in our existing cfg mgmt systems

We would like a way for those host based environment variables to be accessible inside the container used to run each deployment step. With this, we would be able to pass configuration to the deployment step implicitly, simply based on which runner the step was executing.

Without this feature, our options are to create duplicate config inside of BitBucket, however that is inefficient and can also result in having to store/manage values that we would prefer remain inside our environments. Alternatively, we can re-run the same process on each deployment step that we ran when the host was booted to replicate the behaviour and set the env vars inside the step container, but in the context of a fast feedback system, that is somewhat slow.

I guess the 'runner' container would need to be involved given I presume it is responsible for starting the 'deployment step containers' so perhaps being able to pass it a series of env vars with a specific prefix (much like TFVARS in Terraform ) that it will in turn set on the step containers could be an option?

[Norbert Csupka]

Hi @clockworknet 

I appreciate your detailed explanation. Unfortunately this is not available right now, but I opened the following feature request:

BCLOUD-21523 - Allow Bitbucket runner to access host's environmental variable

Our development team will give a first-hand update on that ticket if there's any progress made so I would suggest keeping a watch and vote for it.

Do note however that there's no ETA on enhancement request, and all enhancements are implemented with this policy in mind: Implementation of New Features Policy

Best Regards,
Norbert
Atlassian Bitbucket Cloud Support