How do I properly quote Pipelines pipe variables and escape special characters like dollar signs?

Steven Vaccarella
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 28, 2019

I want to include special characters like quotes, backslashes, dollar signs and newlines in my pipe variables. How do I properly quote and escape them so that they survive processing by the yaml parser and variable expansion?

4 answers

1 accepted

7 votes
Answer accepted
Steven Vaccarella
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 28, 2019

The key to working out how to properly quote and escape pipe variables is to understand that they will undergo two rounds of processing before they make it into the pipe. The first round is yaml parsing. The second round is variable expansion. This can make it particularly tricky to get a special character like a backslash (\) into a pipe variable.

The easiest way to deal with the first round of processing (yaml parsing) is to enclose the entire value of the variable in single quotes. This tells the yaml parser not to process any special characters except the closing single quote character. If you need to include a single quote character in your variable then you just need to double it.

The second round of processing actually happens within your pipeline and is performed by the shell that is running your build script (usually bash). Pipe variables are passed to the pipe using command-line arguments to a "docker run" command. For example, if you define a pipe variable like this in bitbucket-pipelines.yaml:

variables:
MY_PIPE_VARIABLE: 'my-pipe-variable-value'

Then it will get passed into the pipe like this:

docker run ... --env=MY_PIPE_VARIABLE="my-pipe-variable-value" ...

Note the double quotes around the value. This tells your shell to do a limited amount of variable expansion on the value that you've defined. That's what allows you to write something like this in your pipeline:

variables:
MY_PIPE_VARIABLE: '${MY_REPOSITORY_VARIABLE}'

which gets passed into the pipe like this:

docker run ... --env=MY_PIPE_VARIABLE="${MY_REPOSITORY_VARIABLE}" ...

Your shell then replaces ${MY_REPOSITORY_VARIABLE} with the actual value you've defined for that repository variable (in the Pipelines UI).

With this in mind, here are some examples of how to do proper quoting and escaping of pipe variables containing special characters that might otherwise get processed during yaml parsing or variable expansion. In each case I've listed the desired string value first (without any quoting or escaping) followed by a yaml snippet that will produce that value.

 

hello

variables:
VAR1
: 'hello'

 

price: $100

variables:
VAR2
: 'price: \$100'

The dollar sign is escaped to prevent the shell from treating it as a variable expansion.

 

string with internal "double quotes"

variables:
VAR3
: 'string with internal "double quotes"'

This example takes advantage of some special processing for unescaped double quote characters within Pipelines. Pipelines will automatically add a backslash character before any unescaped double quote character to keep your shell happy (without this special processing this example would completely break the "docker run" command - can you see why?).

 

string with internal 'single quotes'

variables:
VAR4
: 'string with internal ''single quotes'''

Single quotes need to be doubled to survive yaml parsing. No further escaping is required for the shell.

 

string with internal `back ticks`

variables:
VAR5
: 'string with internal \`back ticks\`'

We need to escape the back ticks because they are treated specially by the shell, even within double quotes.

 

string with a backslash \ character

variables:
VAR6
: 'string with a backslash \\ character'

Even though we are enclosing our value with single quotes here (which stops the yaml parser from processing backslash characters) we still need to escape the backslash to avoid special processing by the shell in the second round.

 

string with a      tab character and a

newline

variables:
VAR7
: "string with a \ttab character and a \nnewline"

Note here that we're using double quotes around the value instead of single quotes. This tells the yaml parser to expand the "\t" and "\n" sequences into tab and newline characters respectively. The shell doesn't do any further processing on tab or newline characters within double quotes, so no further escaping is required.

 

string with $ multiple ' difficult " characters \ that need

escaping

variables:
VAR8
: "string with \\$ multiple ' difficult \" characters \\\\ that need\nescaping"

Here we've used double quotes around the value so that we can take advantage of yaml expansion of the "\n" sequence into a newline character. However that means that we need extra escaping for backslash and double quote characters. In the case of a backslash we need to escape it twice - once for yaml and once for the shell, so a sequence of four backslash characters will actually end up being just a single backslash by the time your variable makes it all the way into the pipe. On the other hand, the single quote character now doesn't need any escaping at all.

 

string with escaped \t sequences \n that \" remain \' escaped in the \$ final variable inside the pipe

 

variables:
VAR9
: 'string with escaped \\t sequences \\n that \\" remain \\'' escaped in the \\\$ final variable inside the pipe'

 

And finally, here's a repository that uses all these examples in a real pipe. The pipe is designed to just run the "env" command which shows all the final variable values.

https://bitbucket.org/svaccarella/pipe-variable-quoting-example/src/fe9263557c9276d78665558cd1f341bdcf0d035d/bitbucket-pipelines.yml?fileviewer=file-view-default

https://bitbucket.org/svaccarella/pipe-variable-quoting-example/addon/pipelines/home#!/results/1

Steven Vaccarella
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 25, 2019

It's worth mentioning that this answer just describes how to get a particular literal value into the pipe. The pipe is then free to do whatever it wants with that value, including performing an additional round of processing for special characters.

For example see my comment on this related question: https://community.atlassian.com/t5/Bitbucket-Pipelines-questions/old-azcopy-in-Azure-Storage-Deploy-v0-5-0/qaq-p/1012424#qanda-message-1041120

Constantin April 28, 2022
EXTRA_ARGS: '-o ProxyCommand=''ssh -1234 user@host'''

 Sadly, using SSH pipe, this doesn't work. Neither does putting the proxy command (ssh -p1234).

How to solve?

Like Matthias Krams likes this
2 votes
3686 November 1, 2019

Can this be put into official documentation. And if so, can you please link me to said documentation? Thanks!

1 vote
Brad Moulder
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
October 16, 2023

something that saved me a lot of headaches was to just base64 encode the variable. and decode it before usage

1 vote
Marty
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 28, 2019

Hi Steven,

I am not on the Bitbucket team but I will try to help.

Have you tried using backslash to escape all the characters?

Steven Vaccarella
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
February 28, 2019

Hi Martyn,

Actually I am on the Pipelines team. I asked the question with the intention of answering it myself for the benefit of other Community members but it took me a little longer than I expected :)

Like # people like this
Ming Han Chung June 30, 2020

I'm having trouble with a pipe that probably is impacted by this problem. In the pipe.sh I have a command

 

ssh $VARIABLE -i ~/.ssh/ansible_rsa 'sudo su - ansible -c "source /opt/ansible/source-virtualenv.sh && ansible-playbook -i '$VARIABLE1,localhost,' $VARIABLE2 -e 'branch=$BITBUCKET_BRANCH' "' 

 Can you give me any help on how I could fix this?

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events