Way to use "---include-from" in rsync deploy

[robertburczak]

What is the proper way to use "--include-from" argument in rsync deploy?

In my `bitbucket-pipelines.yml` I have added"

EXTRA_ARGS: '--include-from="deploy-include.txt" --exclude="*"'

Of course I have this file in root of my repository.

Unfortunately I am getting this error:

rsync: failed to open include file "./deploy-include.txt": No such file or directory (2)rsync error: error in file IO (code 11) at exclude.c(1207) [client=3.1.3]

[ktomk]

As you have already noticed, a relative path does not cut the deal here.

And also, rsync-deploy (from: Atlassian / Bitbucket Pipelines Pipes) is not overly talkative about the 

EXTRA_ARGS

shell parameter (also known as environment variable).

however the error message you have at hand is from rsync directly.

for trouble-shooting I would first of all suggest to provide the absolute path for

--include-from=

an absolute path is a path that starts with a slash ("/").

It the file is at the root of your repository another shell parameter named BITBUCKET_CLONE_DIR might be of use here. If pipes support variable substitution (which I hope for but I don't know) this would mean that:

EXTRA_ARGS: '--include-from="$BITBUCKET_CLONE_DIR/deploy-include.txt" --exclude="*"'

or

EXTRA_ARGS: '--include-from="${BITBUCKET_CLONE_DIR}/deploy-include.txt" --exclude="*"'

should work.

If variable substitution is not supported here, another option is to hard-encode the value of it. It was just two days ago when I looked last and my pipeline on Atlassian Bitbucket Cloud showed me

BITBUCKET_CLONE_DIR=/opt/atlassian/pipelines/agent/build

so hard-encoding that value would lead to:

EXTRA_ARGS: '--include-from="/opt/atlassian/pipelines/agent/build/deploy-include.txt" --exclude="*"'

To the best of my knowledge any of these three should cut the deal to you.

I would prefer using the standard parameters like BITBUCKET_CLONE_DIR if possible. Albeit I have not seen the value of it changing over the last couple of months, the intention is more clear with the parameter instead of hard-encoding the value it has now.

Hope this helps. If these suggestions give you (different) errors, please share.

[robertburczak]

Thank you @ktomk for your answer. It motivated me to give this problem another try.

You were right about `BITBUCKET_CLONE_DIR`. It's the right path to this file.

But it wasn't problem. It looks like quotation marks shouldn't be used in this env parameter. After all this worked for me:

EXTRA_ARGS: '--include-from=$BITBUCKET_CLONE_DIR/deploy-include.txt --exclude=*'

[ktomk]

Hey @robertburczak thanks for your reply back. That's good to know what you found out. I also found another interesting thing about the EXTRA_ARGS which might be useful here, too: There is an array notation as well:

If you need to use a variable containing a list of values, for example, a list of extra arguments, or an array you can define it directly in the yaml, for example:

variables:
  EXTRA_ARGS: ['--description', 'text containing spaces', '--verbose']

From: Advanced techniques for writing pipes 

This somehow clashes with the description of the pipe which says "string" here, not "array", but perhaps this just works.

I wonder why the pipe is not using the same working directory like the rest of the script. And if it's common across different pipes. I just made a similar suggestion for a just created zip file to be uploaded with a aws beanstalk pipe.

Somewhat curious here.

[robertburczak]

That's even better, but I'll stick with removing quotation marks :)

[ktomk]

Sure, just wanted to tell you what I stumbled over in the meantime. That link also has some notes about the quoting but it could be it's more for writing a pipe on ones own than writing the yaml.

Having the double quotes in the yaml will most likely put them into the arguments verbatim so this would not create the expected result when I was suggesting them to keep the argument as one (e.g. a space in the pathname). This was a bit of a guess by me, I don't really know how the variable substitution is done in the yaml, so our conversation shed some light there.

Thanks again.

[ktomk]

@robertburczak Can you mark the question as answered? This should help others to see it's not yet open. Thanks.