Replacing variables inbetween quotes in a linux command

[Juan Ramirez Uzcategui]

I'm using this command:
- sed -i "s/api_token_placeholder/$PROD_API_KEY/g" backend/file.xml

Where $PROD_API_KEY is a variable configured in bitbucket env variables config...

The key is not being replaced... help!!!

[David Brown]

I ran into the same problem when trying to substitute values into a config file and was able to remedy this by using sed.

sed -i -e 's%{VARIABLE}%'"$SUBSTITUTED_VARIABLE"'%g' filename.json

The key here is breaking the sed command into two chunks using the single quote and the double quote. Once I did this it worked like a charm.

Note: I had to use % instead of the standard / as a delimiter for the sed command as I was also trying to substitute urls.

[Mike Jackson]

This is how we do it for https://battlemaps.co .  Repository variables can be encrypted or unencrypted.  It just seems easier with good old m4.

# install deps
- apt-get update && apt-get install -y m4
- which m4
- echo 'I am a TEST_VAR' | m4 -DTEST_VAR="database name"

- mv src/main/resources/config/application-prod.yml src/main/resources/config/application-prod.m4
- m4 -DM4_TWITTER_CLIENT_ID=${TWITTER_CLIENT_ID} -DM4_TWITTER_CLIENT_SECRET=${TWITTER_CLIENT_SECRET} src/main/resources/config/application-prod.m4>src/main/resources/config/application-prod.yml

Hope it helps!

[Linette]

You might need to escape the variable name with {} (see 

https://stackoverflow.com/questions/17477890/expand-variables-in-sed)

I can’t test that at the mo (replying on a phone) but figured a quicker answer might be of assistance rather till I have my setup in front of me ;) 

[Linette]

Ack, no, that's not it, I just tested in my own pipeline, and the variables replaced fine in my repo, that said it might depend on which docker image you are using.

So first, I'd probably recommend troubleshooting by testing your image and command locally (remember to use -e "PROD_API_KEY=123" to your docker run command if you do this).

Whether you test locally or in bitbucket debugging steps might include adding a few commands like:

cat backend/file.xml (check that it's looking in the right place for the file and can read it)

echo $PROD_API_KEY (make sure the value is readable)

hope that helps!

[Juan Ramirez Uzcategui]

Hi Linette, thanks for your answer, but if I hardcode the API key it works. If I use the variable in other places it's replaced correctly.

Does having the key as an encrypted key has anything to do with whats happening?

Thanks,

[Linette]

Ooohh... mysterious!

It does sound like something odd is happening with the sed command itself.  Ooooor, it might also be something odd happening with quoting elsewhere in your script having a knock on effect.

Maybe try: *fingers crossed*

- sed -i 's/api_token_placeholder/'"$PROD_API_KEY"'/g' backend/file.xml

Note that I've closed the single quotes before double quoting the variable, and then re-opened them for the /g.

If this doesn't work, I have some questions:

When you say encrypted do you mean a secure variable (which should have no difference) or base64 encoded like we do for SSH here ? If the later, you'll have to decode the encryption.

And does it not work at all? or is your api_token_placeholder replaced with the literal string of '$PROD_API_KEY'?

[Juan Ramirez Uzcategui]

ha! Testing it like this on a mac os or linux terminal doesn't work...

PROD_API_KEY=blabla sed "s/api_token_placeholder/$PROD_API_KEY/g" backend/file.xml

I've tried all the different combinations... 

And answering your question, it was encrypted as a secure variable.

Thanks!

[Linette]

Yeah, a direct mac or linux terminal will not quite behave in the same way as a docker container, which is why we recommend when testing locally you use docker (https://confluence.atlassian.com/x/IQr3MQ)

and how about my second question:

And does it not work at all? or is your api_token_placeholder replaced with the literal string of '$PROD_API_KEY'?

That said, it might be time to open a case with support, they could work realtime with you. I imagine you must be quite frustrated with the delay by now!