bitbucket pipeline leaks secrets

[Nick de Palézieux]

We're using the ssh-run pipe to execute a build step on a remote machine.

This build step needs access to secrets which we have defined as "secured" bitbucket workspace variables.

When running the ssh-run pipe, the content of one of those variables is printed in plain text, however.

This is what the relevant parts of our bitbucket-pipelines.yml looks like:

image: atlassian/default-image:2

pipelines:
  branches:
    develop:
        - step:
            name: Deploy to test
            deployment: Test
            script:
              - pipe: atlassian/ssh-run:0.4.2
                variables:
                SSH_USER: $BUILD_USER
                SERVER: $BUILD_SERVER
                PORT: $BUILD_SSH_PORT
                SSH_KEY: $BITBUCKET_CI_SSH_KEY
                MODE: "script"
                COMMAND: "tools/ci-deploy.sh"
                ENV_VARS: >-
                  GCP_WRITER_JSON='${GCP_WRITER_JSON}'
                  REMOTE_CI_SSH_KEY='${REMOTE_CI_SSH_KEY}'

During setup of the step, the command

docker container run ... bitbucketpipelines/ssh-run:0.4.2

does not reveal the content of the secret variables.

But when that pipe executes the actual SSH command

ssh -i /root/.ssh/pipelines_id ... GCP_WRITER_JSON='<THE CONTENT IN PLAIN TEXT>' REMOTE_CI_SSH_KEY='<THE CONTENT IS HIDDEN>'

the variable content is visible in the pipeline log.

This is problematic. Can this be avoided by defining the pipeline differently?

EDIT:

I suspect this has something to do with the content of the GCP_WRITER_JSON variable. It looks as follows

{ "type": "service_account", "project_id": "my-container-registry", "private_key_id": "...", "private_key": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----\n", "client_email": "...@my-container-registry.iam.gserviceaccount.com", "client_id": "...", "auth_uri": "...", "token_uri": "...", "auth_provider_x509_cert_url": "...", "client_x509_cert_url": "...",}

[Aron Gombas _Midori_ · Accepted solution]

According to the doc, "secured" Pipeline variables should be printed in "masked" format.

If try to print it out with "echo", is it shown as "masked" or the real value?

[Nick de Palézieux]

It works in other places, it's just the `ssh` command where it seems to fail.

[Aron Gombas]

In that case what you're doing is correct and it is probably not a bug in Bitbucket, either, but I'd think that this pipe probably doesn't respect that this should not be printed:

https://bitbucket.org/atlassian/ssh-run/src/master/pipe/pipe.sh

Which depends on this:

https://bitbucket.org/bitbucketpipelines/bitbucket-pipes-toolkit-bash/raw/0.6.0/common.sh

I'd check if it is possible to rewrite the console prints in these scripts. But this is just a hint, someone may come up with a more precise answer.

[Nick de Palézieux]

I checked again and it turns out the problem is only present for one of the two secured variables. It might be related to the fact that this variable's content is JSON, so the masking logic gets confused?

[Aron Gombas _Midori_]

I cannot know, sorry.

But when you originally described the problem, my first idea was "isn't this possible to pass that variable in an alternative way? like saving this to temp file and referencing the file? or some other way which will never be printed in plain text?"

[Nick de Palézieux]

Thanks for your help anyway.

I think the correct solution would be to store those secrets on the remote machine s.t. the pipe does not have to pass them over SSH and doesn't need to reference them at all.

I was trying to minimize dependencies/setup requirements for the remote machine, but this seems to come at the cost of security.

[Patrik S]

Hello @Nick de Palézieux ,

I think your assumption is correct, the content of the variable being a JSON file might be confusing the masking algorithm. I suspect that the pipe is doing some sort of escaping in the content of the JSON file before including it in the final SSH command, which is likely making that particular line in the logs don't exactly match the value you have configured in the env variable, so the masking algorithm is not replacing it with the variable name. This would explain why it just happen with the JSON variable, but not with the other variable.

One workaround I can think of is to base64 encode the value of the JSON file and then save the encoded value in the environment variable on Bitbucket side. Then, when running the scripts on your remote host, you could add a command to base64 to decode the variable in run time before using its value.

If you are using Linux, the below command can be used to base64 encode the content of a file: 

base64 secrets.json

 and to decode the content of an env variable : 

echo $MY_ENCODED_SECRET | base64 --decode

Thank you, @Nick de Palézieux !

Patrik S

[Nick de Palézieux]

Thanks for your response, @Patrik S .

I will try base64-encoding.

Regarding your suspicion that "the pipe is doing some sort of escaping in the content of the JSON file before including it in the final SSH command", I can rule that out since a simple

script:
  - echo $GCP_WRITER_JSON

also leaks the secret. So it doesn't appear to be related to the ssh-pipe. I changed the title of the post to reflect this.

[Patrik S]

Hello @Nick de Palézieux 

Thank you for the confirmation and for testing outside of the pipe as well.

I did some tests on my end and was able to reproduce this behavior only when the content of the variable included new line characters('\n').

Pipeline variables don't support newline characters, so if the content you copied/pasted when setting the value to the secure variable did contain newline characters (\n), the masking algorithm will not be able to hide it from the logs.

In this case, you need to make sure the content of the variable is contained in a single line. During my tests, after removing all the new line characters of the JSON file, the content was correctly masked in the logs.

If you are using Linux, the following command can be used to remove all newline characters from the content of a file : 

tr -d '\n' < my_file.json

The other alternative, as previously mentioned, is to base64 encode the content prior to saving it to the variable, and then decode it when the value will be used.

Thank you, @Nick de Palézieux !

Patrik S

[Nick de Palézieux]

You're right, the variable does contain newline characters! Thanks for figuring this out.

I will change it to be base64 encoded or figure out a way where it doesn't need to be transmitted at all.

I do think it would be good to document this limitation in the documentation.

[Patrik S]

Awesome! 

I agree with you this info should be included in the docs, so I will reach out internally to have it updated :) 

Thank you, @Nick de Palézieux !

Patrik S