It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How do I properly quote Pipelines pipe variables and escape special characters like dollar signs?

I want to include special characters like quotes, backslashes, dollar signs and newlines in my pipe variables. How do I properly quote and escape them so that they survive processing by the yaml parser and variable expansion?

2 answers

1 accepted

2 votes
Answer accepted

The key to working out how to properly quote and escape pipe variables is to understand that they will undergo two rounds of processing before they make it into the pipe. The first round is yaml parsing. The second round is variable expansion. This can make it particularly tricky to get a special character like a backslash (\) into a pipe variable.

The easiest way to deal with the first round of processing (yaml parsing) is to enclose the entire value of the variable in single quotes. This tells the yaml parser not to process any special characters except the closing single quote character. If you need to include a single quote character in your variable then you just need to double it.

The second round of processing actually happens within your pipeline and is performed by the shell that is running your build script (usually bash). Pipe variables are passed to the pipe using command-line arguments to a "docker run" command. For example, if you define a pipe variable like this in bitbucket-pipelines.yaml:

variables:
MY_PIPE_VARIABLE: 'my-pipe-variable-value'

Then it will get passed into the pipe like this:

docker run ... --env=MY_PIPE_VARIABLE="my-pipe-variable-value" ...

Note the double quotes around the value. This tells your shell to do a limited amount of variable expansion on the value that you've defined. That's what allows you to write something like this in your pipeline:

variables:
MY_PIPE_VARIABLE: '${MY_REPOSITORY_VARIABLE}'

which gets passed into the pipe like this:

docker run ... --env=MY_PIPE_VARIABLE="${MY_REPOSITORY_VARIABLE}" ...

Your shell then replaces ${MY_REPOSITORY_VARIABLE} with the actual value you've defined for that repository variable (in the Pipelines UI).

With this in mind, here are some examples of how to do proper quoting and escaping of pipe variables containing special characters that might otherwise get processed during yaml parsing or variable expansion. In each case I've listed the desired string value first (without any quoting or escaping) followed by a yaml snippet that will produce that value.

 

hello

variables:
VAR1
: 'hello'

 

price: $100

variables:
VAR2
: 'price: \$100'

The dollar sign is escaped to prevent the shell from treating it as a variable expansion.

 

string with internal "double quotes"

variables:
VAR3
: 'string with internal "double quotes"'

This example takes advantage of some special processing for unescaped double quote characters within Pipelines. Pipelines will automatically add a backslash character before any unescaped double quote character to keep your shell happy (without this special processing this example would completely break the "docker run" command - can you see why?).

 

string with internal 'single quotes'

variables:
VAR4
: 'string with internal ''single quotes'''

Single quotes need to be doubled to survive yaml parsing. No further escaping is required for the shell.

 

string with internal `back ticks`

variables:
VAR5
: 'string with internal \`back ticks\`'

We need to escape the back ticks because they are treated specially by the shell, even within double quotes.

 

string with a backslash \ character

variables:
VAR6
: 'string with a backslash \\ character'

Even though we are enclosing our value with single quotes here (which stops the yaml parser from processing backslash characters) we still need to escape the backslash to avoid special processing by the shell in the second round.

 

string with a      tab character and a

newline

variables:
VAR7
: "string with a \ttab character and a \nnewline"

Note here that we're using double quotes around the value instead of single quotes. This tells the yaml parser to expand the "\t" and "\n" sequences into tab and newline characters respectively. The shell doesn't do any further processing on tab or newline characters within double quotes, so no further escaping is required.

 

string with $ multiple ' difficult " characters \ that need

escaping

variables:
VAR8
: "string with \\$ multiple ' difficult \" characters \\\\ that need\nescaping"

Here we've used double quotes around the value so that we can take advantage of yaml expansion of the "\n" sequence into a newline character. However that means that we need extra escaping for backslash and double quote characters. In the case of a backslash we need to escape it twice - once for yaml and once for the shell, so a sequence of four backslash characters will actually end up being just a single backslash by the time your variable makes it all the way into the pipe. On the other hand, the single quote character now doesn't need any escaping at all.

 

string with escaped \t sequences \n that \" remain \' escaped in the \$ final variable inside the pipe

 

variables:
VAR9
: 'string with escaped \\t sequences \\n that \\" remain \\'' escaped in the \\\$ final variable inside the pipe'

 

And finally, here's a repository that uses all these examples in a real pipe. The pipe is designed to just run the "env" command which shows all the final variable values.

https://bitbucket.org/svaccarella/pipe-variable-quoting-example/src/fe9263557c9276d78665558cd1f341bdcf0d035d/bitbucket-pipelines.yml?fileviewer=file-view-default

https://bitbucket.org/svaccarella/pipe-variable-quoting-example/addon/pipelines/home#!/results/1

It's worth mentioning that this answer just describes how to get a particular literal value into the pipe. The pipe is then free to do whatever it wants with that value, including performing an additional round of processing for special characters.

For example see my comment on this related question: https://community.atlassian.com/t5/Bitbucket-Pipelines-questions/old-azcopy-in-Azure-Storage-Deploy-v0-5-0/qaq-p/1012424#qanda-message-1041120

1 vote

Hi Steven,

I am not on the Bitbucket team but I will try to help.

Have you tried using backslash to escape all the characters?

Hi Martyn,

Actually I am on the Pipelines team. I asked the question with the intention of answering it myself for the benefit of other Community members but it took me a little longer than I expected :)

Like Martyn Winsen likes this

Suggest an answer

Log in or Sign up to answer
Community showcase
Published in Bitbucket Pipelines

Building a Bitbucket Pipe as a casual coder

...ipe.sh :  #!/bin/bash source "$(dirname "$0")/common.sh" enable_debug extra_args="" if [[ "${DEBUG}" == "true" ]]; then extra_args="--verbose" fi # mandatory variables R...

4,038 views 4 22
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you