The key to working out how to properly quote and escape pipe variables is to understand that they will undergo two rounds of processing before they make it into the pipe. The first round is yaml parsing. The second round is variable expansion. This can make it particularly tricky to get a special character like a backslash (\) into a pipe variable.
The easiest way to deal with the first round of processing (yaml parsing) is to enclose the entire value of the variable in single quotes. This tells the yaml parser not to process any special characters except the closing single quote character. If you need to include a single quote character in your variable then you just need to double it.
The second round of processing actually happens within your pipeline and is performed by the shell that is running your build script (usually bash). Pipe variables are passed to the pipe using command-line arguments to a "docker run" command. For example, if you define a pipe variable like this in bitbucket-pipelines.yaml:
variables:
MY_PIPE_VARIABLE: 'my-pipe-variable-value'
Then it will get passed into the pipe like this:
docker run ... --env=MY_PIPE_VARIABLE="my-pipe-variable-value" ...
Note the double quotes around the value. This tells your shell to do a limited amount of variable expansion on the value that you've defined. That's what allows you to write something like this in your pipeline:
variables:
MY_PIPE_VARIABLE: '${MY_REPOSITORY_VARIABLE}'
which gets passed into the pipe like this:
docker run ... --env=MY_PIPE_VARIABLE="${MY_REPOSITORY_VARIABLE}" ...Your shell then replaces ${MY_REPOSITORY_VARIABLE} with the actual value you've defined for that repository variable (in the Pipelines UI).
With this in mind, here are some examples of how to do proper quoting and escaping of pipe variables containing special characters that might otherwise get processed during yaml parsing or variable expansion. In each case I've listed the desired string value first (without any quoting or escaping) followed by a yaml snippet that will produce that value.
hello
variables:
VAR1: 'hello'
price: $100
variables:
VAR2: 'price: \$100'
The dollar sign is escaped to prevent the shell from treating it as a variable expansion.
string with internal "double quotes"
variables:
VAR3: 'string with internal "double quotes"'
This example takes advantage of some special processing for unescaped double quote characters within Pipelines. Pipelines will automatically add a backslash character before any unescaped double quote character to keep your shell happy (without this special processing this example would completely break the "docker run" command - can you see why?).
string with internal 'single quotes'
variables:
VAR4: 'string with internal ''single quotes'''
Single quotes need to be doubled to survive yaml parsing. No further escaping is required for the shell.
string with internal `back ticks`
variables:
VAR5: 'string with internal \`back ticks\`'
We need to escape the back ticks because they are treated specially by the shell, even within double quotes.
string with a backslash \ character
variables:
VAR6: 'string with a backslash \\ character'
Even though we are enclosing our value with single quotes here (which stops the yaml parser from processing backslash characters) we still need to escape the backslash to avoid special processing by the shell in the second round.
string with a tab character and a
newline
variables:
VAR7: "string with a \ttab character and a \nnewline"
Note here that we're using double quotes around the value instead of single quotes. This tells the yaml parser to expand the "\t" and "\n" sequences into tab and newline characters respectively. The shell doesn't do any further processing on tab or newline characters within double quotes, so no further escaping is required.
string with $ multiple ' difficult " characters \ that need
escaping
variables:
VAR8: "string with \\$ multiple ' difficult \" characters \\\\ that need\nescaping"
Here we've used double quotes around the value so that we can take advantage of yaml expansion of the "\n" sequence into a newline character. However that means that we need extra escaping for backslash and double quote characters. In the case of a backslash we need to escape it twice - once for yaml and once for the shell, so a sequence of four backslash characters will actually end up being just a single backslash by the time your variable makes it all the way into the pipe. On the other hand, the single quote character now doesn't need any escaping at all.
string with escaped \t sequences \n that \" remain \' escaped in the \$ final variable inside the pipe
variables:
VAR9: 'string with escaped \\t sequences \\n that \\" remain \\'' escaped in the \\\$ final variable inside the pipe'
And finally, here's a repository that uses all these examples in a real pipe. The pipe is designed to just run the "env" command which shows all the final variable values.
https://bitbucket.org/svaccarella/pipe-variable-quoting-example/src/fe9263557c9276d78665558cd1f341bdcf0d035d/bitbucket-pipelines.yml?fileviewer=file-view-default
https://bitbucket.org/svaccarella/pipe-variable-quoting-example/addon/pipelines/home#!/results/1
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.