It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

aws-s3-deploy updates unchanged files

We use the aws-s3-deploy pipe with the following configuration to upload files from the repository to s3. 

-   pipe: atlassian/aws-s3-deploy:0.2.1
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID_STAGE
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY_STAGE
AWS_DEFAULT_REGION: eu-central-1
S3_BUCKET: my-bucket
LOCAL_PATH: resources
DELETE_FLAG: 'true'

Even though no of the files in `resources` has changed, the entire folder is uploaded on each deployment.

The files in the given local path are commited in the repository and should also not be affected by any other steps in the deployment.

1 answer

0 votes

Hi @Nicolas . The pipe users the s3 sync CLI under the hood, so it's a bit tricky to sync only changed files. One possible solution might be to use the combination of --exclude/--include options (see the sync command reference for details). You can pass them via the EXTRA_ARGS pipe variable. Basically, you could get the list of changed files from git and include these files and exclude files that were unchanged. 

thanks for the insights. I was confused by the description in the readme

Recursively copies new and updated files from the source local directory to the destination.

maybe this part should be removed then?

@Nicolas you could also try to add --size-only to the EXTRA_ARGS pipe variable. This will tell the aws s3 sync to only upload files that have changed the size. See https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html for how this option affects the sync command behavior.

your suggested approach wouldn't work as the size can remain unchanged even though the file content has been changed. Please correct me if I am wrong.
I do have the same issue and want to upload/sync only the changed files to S3.

i have the same problem as this, did anyone find a solution to it?  also need the same for the google bucket upload as well

 

 

would, this work?

--exclude --exact-timestamps

 

how about this, can this be used incorporated?https://github.com/MathieuLoutre/grunt-aws-s3

 

the 

options.differential

@davidstanton  try the --size-only if you want to sync affected files only , it works. Adding --size-only won't make the entire repo to be synchronised again, but only latest changed files

Like davidstanton likes this

even if file size stays the same and only the contents of the file changed?

i know it doesnt make any sense .. but if the size stays the same and file was changed, then it is going to be "indetified" and copied over s3 .. only that file

Like davidstanton likes this

brilliant, thanks.  Now, any clue how to do the same with a google bucket? lol

sadly I haven't had any experiences with gcp

ok, np.  how about invalidating AWS cloudfront file, based on what 'new" file was uploaded to the SS bucket, rather than at the moment, im invalidating /* (all basically)

this --size-only actually doesn't work, i've just had an instance where the content changed but the size didn't and it didn't upload the file with the new changes

Was there any update to this cause I also did try --size-only and it doesn't work!

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket Pipelines

What We Learned When We Researched Open Source Vulnerabilities in 7 Popular Coding Languages

...hey are a part of us, shaping how we interact with the world around us. The same holds true for programming languages when we think about how different kinds of vulnerabilities raise their heads in t...

1,034 views 0 3
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you