aws-s3-deploy updates unchanged files

[Nicolas]

We use the aws-s3-deploy pipe with the following configuration to upload files from the repository to s3. 

-   pipe: atlassian/aws-s3-deploy:0.2.1
    variables:
        AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID_STAGE
        AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY_STAGE
        AWS_DEFAULT_REGION: eu-central-1
        S3_BUCKET: my-bucket
        LOCAL_PATH: resources
        DELETE_FLAG: 'true'

Even though no of the files in `resources` has changed, the entire folder is uploaded on each deployment.

The files in the given local path are commited in the repository and should also not be affected by any other steps in the deployment.

[Alexander Zhukov]

Hi @Nicolas . The pipe users the s3 sync CLI under the hood, so it's a bit tricky to sync only changed files. One possible solution might be to use the combination of --exclude/--include options (see the sync command reference for details). You can pass them via the EXTRA_ARGS pipe variable. Basically, you could get the list of changed files from git and include these files and exclude files that were unchanged. 

[Nicolas]

thanks for the insights. I was confused by the description in the readme

Recursively copies new and updated files from the source local directory to the destination.

maybe this part should be removed then?

[Alexander Zhukov]

@Nicolas you could also try to add --size-only to the EXTRA_ARGS pipe variable. This will tell the aws s3 sync to only upload files that have changed the size. See https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html for how this option affects the sync command behavior.

[naspar]

your suggested approach wouldn't work as the size can remain unchanged even though the file content has been changed. Please correct me if I am wrong.
I do have the same issue and want to upload/sync only the changed files to S3.

[davidstanton]

i have the same problem as this, did anyone find a solution to it?  also need the same for the google bucket upload as well

would, this work?

--exclude --exact-timestamps

[davidstanton]

how about this, can this be used incorporated?https://github.com/MathieuLoutre/grunt-aws-s3

the 

options.differential

[naspar]

@davidstanton  try the --size-only if you want to sync affected files only , it works. Adding --size-only won't make the entire repo to be synchronised again, but only latest changed files

[davidstanton]

even if file size stays the same and only the contents of the file changed?

[naspar]

i know it doesnt make any sense .. but if the size stays the same and file was changed, then it is going to be "indetified" and copied over s3 .. only that file

[davidstanton]

brilliant, thanks.  Now, any clue how to do the same with a google bucket? lol

[naspar]

sadly I haven't had any experiences with gcp

[davidstanton]

ok, np.  how about invalidating AWS cloudfront file, based on what 'new" file was uploaded to the SS bucket, rather than at the moment, im invalidating /* (all basically)

[davidstanton]

this --size-only actually doesn't work, i've just had an instance where the content changed but the size didn't and it didn't upload the file with the new changes

[johndheem NA]

Was there any update to this cause I also did try --size-only and it doesn't work!