Exclude files from upload to s3 bucket using "Amazon s3 object task"

[bugretvup]

Hi,

I'm trying to replace "aws s3 sync" script task with "Amazon S3 Object" task, on a server install of bamboo  (v6.8.1),  in a build plan that uploads content to a s3 bucket.

Trouble i'm facing right now are two:

1) i need to exclude some files from the upload process.

I've tried to get it working using Ant exclude patterns (checking the option "Use Ant patterns to select files"), but with no positive outcome until now. 

2) if problem (1) get's solved, and if possible, i would also like to have a 'sync' approach, that is, that the execution of the plan, also removes objects on s3 bucket if they're not present in the local folder anymore.

Documentation i've tried following:

- https://utoolity.atlassian.net/wiki/spaces/TAWS/pages/19464196/Using+the+Amazon+S3+Object+task+in+Bamboo

- http://ant.apache.org/manual/Tasks/defaultexcludes.html

HAs anyone got this to work?

Thanks

-wm

[Steffen Opel _Utoolity_]

Hi Werner,

Thank you for using our apps, and sorry for the trouble. I'm afraid your requirements are simply not supported at this point:

1. The 'Use Ant patterns to select files' option currently only supports inclusion of files based on the pattern syntax itself, but not the explicit exclusion of files via any of the 'exclude' XML tags.

• As a potential workaround, you might want to consider assembling the desired fileset in a new directory, which would allow to use resp. OS level tooling on copy to exclude parts of the source directory hierarchy (specifics depend on the OS and shell in question).
• I realize that the option's description and linked documentation from Ant are misleading in this regard and we will try to change it so that future users are less likely to run into this, thanks for pointing out this usability flaw.

2. The Amazon S3 Object task does indeed not yet support an equivalent for the aws . s3 . sync action, and we still recommend falling back to the AWS CLI as a workaround for such scenarios accordingly. Please watch and vote for the following story to guide our roadmap:

• As a user, I want a task to sync files with S3 so that upload/download bandwith/time is minimized (UAA-288)

I hope this helps despite lacking an actual solution.

Cheers,
Steffen

[bugretvup]

Hi Steffen

Thank you for your clarifying answer.

I think that for the time being i'll go back to a shell/system 'aws s3 sync' for this task, at least until UAA-288 is a reality.

Have a nice week

-wm

[bugretvup]

Hi CCM,

Sorry, you're right, should have included that information. 

Let's say i have a local directory (from a git cloned on previous task in the plan) with:

./file1.ext

./file2.ext

/dir1/file1-dir1.ext

/dir-dont-upload/file1-dir-dont-upload.ext

./file3.ext
./.some_module_rc

My "Amazon S3 Object" task is configured to upload the content of this folder to a s3 bucket.

I've checked the "[ x ]Use Ant patterns to select files" option, and in "Source Local Path" field i've tried a lot of combinations, right now i'm using.

**,<defaultexcludes echo="true" add="/dir-dont-upload/**"/>, 

i've also tried 

**,<exclude add="/dir-dont-upload/**"/>, 

But it always uploads all the files.

If i remove the "**" starting, it doesn't upload anything.

Also, is there a quicker way (like some sandboxes commandline with debug, i could use to test this, as always run the entire plan, is somewhat complicated.

Thanks for any suggestion.

-wm

[Craig Castle-Mead]

Hi Werner,

Are you able to share the ant pattern you’re using, what result you’re expecting and the result you’re getting? This may be enough for the community to help diagnose and suggest a fix.

CCM