only upload changed files to AWS S3?

[davidstanton]

hi

would anyone know how i can only upload the changed files to aws s3 using:

https://bitbucket.org/atlassian/aws-s3-deploy/src/master/README.md

at the moment it seems to upload every file, so the pipline takes much longer to run than needs be.

[Oleksandr Kyrdan]

Hi @davidstanton  ,

Thank you for the question!

The aws-s3-deploy pipe uses the aws s3 sync .

It syncs directories and S3 prefixes. Recursively copies new and updated files from the source local directory to the destination. Only creates folders in the destination if they contain one or more files.  

If files stored in your S3 bucket, the pipe uploads only changed files.

Cheer's,
Alex

[davidstanton]

is this an update then?, as V 0.4.1  didn't upload only changed files as I was using: EXTRA_ARGS: '--size-only'    to try and get around it, but that only worked half the time, as some files don't change size, even though made changes in them

[davidstanton]

nope, this still don't work as expected, it still uploads all the files and not syncs the changed files, so its worse in effect than using EXTRA_ARGS: '--size-only'  which sometimes doesn't upload a file if its size doesn't change ( adding a few return lines at the bottom of a files, sort of gets around this, but it ain't really ideal)

[davidstanton]

there is also another step on this as well, if its claiming to sync only changed files and also suggests to use cache invalidation with it,  it should only then invalidate the cache for the uploaded changed files,  new files just get cached in time anyway.  Invalidating "All" which it currently does isn't a problem in itself but its the subsequent pulls that cloudfront then does, as it will pull "all" files back to cache, which costs more money etc in cloudfront charges.  No point invalidating files which don't need to be, when this latest 0.4.3 pipe "claims" to sync only updated files