bitbucket pipeline leaks secrets

Nick de Palézieux May 10, 2023

We're using the ssh-run pipe to execute a build step on a remote machine.

This build step needs access to secrets which we have defined as "secured" bitbucket workspace variables.

When running the ssh-run pipe, the content of one of those variables is printed in plain text, however.

 

This is what the relevant parts of our bitbucket-pipelines.yml looks like:

image: atlassian/default-image:2

pipelines:
branches:
develop:
- step:
name: Deploy to test
deployment: Test
script:
- pipe: atlassian/ssh-run:0.4.2
variables:
SSH_USER: $BUILD_USER
SERVER: $BUILD_SERVER
PORT: $BUILD_SSH_PORT
SSH_KEY: $BITBUCKET_CI_SSH_KEY
MODE: "script"
COMMAND: "tools/ci-deploy.sh"
ENV_VARS: >-
GCP_WRITER_JSON='${GCP_WRITER_JSON}'
REMOTE_CI_SSH_KEY='${REMOTE_CI_SSH_KEY}'

 

During setup of the step, the command

docker container run ... bitbucketpipelines/ssh-run:0.4.2

does not reveal the content of the secret variables.

 

But when that pipe executes the actual SSH command

ssh -i /root/.ssh/pipelines_id ... GCP_WRITER_JSON='<THE CONTENT IN PLAIN TEXT>' REMOTE_CI_SSH_KEY='<THE CONTENT IS HIDDEN>'

the variable content is visible in the pipeline log.

 

This is problematic. Can this be avoided by defining the pipeline differently?

 

EDIT:

I suspect this has something to do with the content of the GCP_WRITER_JSON variable. It looks as follows

{ "type": "service_account", "project_id": "my-container-registry", "private_key_id": "...", "private_key": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----\n", "client_email": "...@my-container-registry.iam.gserviceaccount.com", "client_id": "...", "auth_uri": "...", "token_uri": "...", "auth_provider_x509_cert_url": "...", "client_x509_cert_url": "...",}

 

1 answer

1 accepted

0 votes
Answer accepted
Aron Gombas _Midori_
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
May 10, 2023

According to the doc, "secured" Pipeline variables should be printed in "masked" format.

If try to print it out with "echo", is it shown as "masked" or the real value?

Nick de Palézieux May 10, 2023

It works in other places, it's just the `ssh` command where it seems to fail.

Aron Gombas
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
May 10, 2023

In that case what you're doing is correct and it is probably not a bug in Bitbucket, either, but I'd think that this pipe probably doesn't respect that this should not be printed:

https://bitbucket.org/atlassian/ssh-run/src/master/pipe/pipe.sh

Which depends on this:

https://bitbucket.org/bitbucketpipelines/bitbucket-pipes-toolkit-bash/raw/0.6.0/common.sh

I'd check if it is possible to rewrite the console prints in these scripts. But this is just a hint, someone may come up with a more precise answer.

Nick de Palézieux May 11, 2023

I checked again and it turns out the problem is only present for one of the two secured variables. It might be related to the fact that this variable's content is JSON, so the masking logic gets confused?

Aron Gombas _Midori_
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
May 11, 2023

I cannot know, sorry.

But when you originally described the problem, my first idea was "isn't this possible to pass that variable in an alternative way? like saving this to temp file and referencing the file? or some other way which will never be printed in plain text?"

Nick de Palézieux May 11, 2023

Thanks for your help anyway.

I think the correct solution would be to store those secrets on the remote machine s.t. the pipe does not have to pass them over SSH and doesn't need to reference them at all.

 

I was trying to minimize dependencies/setup requirements for the remote machine, but this seems to come at the cost of security.

Like Aron Gombas _Midori_ likes this
Patrik S
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
May 11, 2023

Hello @Nick de Palézieux ,

I think your assumption is correct, the content of the variable being a JSON file might be confusing the masking algorithm. I suspect that the pipe is doing some sort of escaping in the content of the JSON file before including it in the final SSH command, which is likely making that particular line in the logs don't exactly match the value you have configured in the env variable, so the masking algorithm is not replacing it with the variable name. This would explain why it just happen with the JSON variable, but not with the other variable.

One workaround I can think of is to base64 encode the value of the JSON file and then save the encoded value in the environment variable on Bitbucket side. Then, when running the scripts on your remote host, you could add a command to base64 to decode the variable in run time before using its value.

If you are using Linux, the below command can be used to base64 encode the content of a file: 

base64 secrets.json

 and to decode the content of an env variable : 

echo $MY_ENCODED_SECRET | base64 --decode

Thank you, @Nick de Palézieux !

Patrik S

Nick de Palézieux May 11, 2023

Thanks for your response, @Patrik S .

 

I will try base64-encoding.

 

Regarding your suspicion that "the pipe is doing some sort of escaping in the content of the JSON file before including it in the final SSH command", I can rule that out since a simple

script:
- echo $GCP_WRITER_JSON

also leaks the secret. So it doesn't appear to be related to the ssh-pipe. I changed the title of the post to reflect this.

Patrik S
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
May 11, 2023

Hello @Nick de Palézieux 

Thank you for the confirmation and for testing outside of the pipe as well.

I did some tests on my end and was able to reproduce this behavior only when the content of the variable included new line characters('\n').

Pipeline variables don't support newline characters, so if the content you copied/pasted when setting the value to the secure variable did contain newline characters (\n), the masking algorithm will not be able to hide it from the logs.

In this case, you need to make sure the content of the variable is contained in a single line. During my tests, after removing all the new line characters of the JSON file, the content was correctly masked in the logs.

If you are using Linux, the following command can be used to remove all newline characters from the content of a file : 

tr -d '\n' < my_file.json

The other alternative, as previously mentioned, is to base64 encode the content prior to saving it to the variable, and then decode it when the value will be used.

Thank you, @Nick de Palézieux !

Patrik S

Nick de Palézieux May 12, 2023

You're right, the variable does contain newline characters! Thanks for figuring this out.

I will change it to be base64 encoded or figure out a way where it doesn't need to be transmitted at all.

 

I do think it would be good to document this limitation in the documentation.

Like Patrik S likes this
Patrik S
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
May 12, 2023

Awesome! 

I agree with you this info should be included in the docs, so I will reach out internally to have it updated :) 

Thank you, @Nick de Palézieux !

Patrik S

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
AUG Leaders

Atlassian Community Events