aws-s3-deploy 1.0 break upload (not happening in 0.5.0)

Question

We have a pipeline that send our website build to a S3 bucket to serve files through CloudFront.

Today we tried to update our pipeline script to aws-s3-deploy 1.0 but this break the pipeline without a meaninful error.

Reverting to 0.5.0 fix the issue.

Here there is the interesting part of our pipeline script:

develop:
      - step:
          name: Build
          caches:
            - node
          script:
            - npm install
            - npm run build:devel
          artifacts:
            - home/**
      - step:
          name: Deploy to S3
          deployment: Development
          script:
            - pipe: atlassian/aws-s3-deploy:1.0.0
              variables:
                S3_BUCKET: ${S3_BUCKET}/home
                LOCAL_PATH: 'home'
                ACL: 'public-read'
                EXTRA_ARGS: '--exclude=*.html'
            - pipe: atlassian/aws-s3-deploy:1.0.0
              variables:
                S3_BUCKET: ${S3_BUCKET}/home
                LOCAL_PATH: 'home'
                ACL: 'public-read'
                CACHE_CONTROL: 'max-age=600'
                EXTRA_ARGS: '--exclude=* --include=*.html'
            - pipe: atlassian/aws-s3-deploy:1.0.0
              variables:
                S3_BUCKET: ${S3_BUCKET}
                LOCAL_PATH: 'home'
                ACL: 'public-read'
                CACHE_CONTROL: 'max-age=600'
                EXTRA_ARGS: '--exclude=* --include=index.html'

And the IAM policy attached to the user on AWS:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${S3_BUCKET}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTagging",
                "s3:HeadObject",
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:PutObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::${S3_BUCKET}/*"
            ]
        }
    ]
}

Here (on Dropbox due to post characters limit) there is a full log of the upload step that fails with DEBUG: 'true' option set:

https://www.dropbox.com/s/aogfbqw86vy3su2/pipelineLog-3.txt?dl=0

Leonardo Medici · Answer

@Oleksandr KyrdanI was looking some minutes ago at your repository and found the note in the latest commits.

I also found out that some of our pdf files have non utf-8 charaters, probably the root cause was a copy-paste from MS-Word that use non standard charaters for ' " - (at least in my native language)

Changed the characters and now it's working like a charm.

Thank you for your help

Oleksandr Kyrdan · Answer

Hi @Leonardo Medici

Some summary after the issue investigation,

Started from version 1.0.0 aws-s3-deploy pipe uses AWS CLI version 2.

So, now AWS CLI version 2 more strict for filenames/key-names: only utf-8 supported.

So, the official AWS S3 Creating object key names guide should be updated by AWS to the next:

Object key naming guidelines

You can must use any UTF-8 character in an object key name. However, using certain characters in key names can cause problems with some applications and protocols. The following guidelines help you maximize compliance with DNS, web-safe characters, XML parsers, and other APIs.

To solve your issue, try not to use non-utf-8 character in an object key name (filename).

According to your question, we updated Readme for the  pipe and released new version

- pipe: atlassian/aws-s3-deploy:1.0.1

Thank you for your contribution!

Cheers,
OIeksandr Kyrdan

Leonardo Medici · Answer

@Oleksandr KyrdanI tried to run the pipeline with AWS_CLI_FILE_ENCODING variable without success.

I don't know if object key is an issues, I tried to manually run S3 sync command with

aws-cli/2.2.15 Python/3.9.5 Darwin/20.5.0 source/x86_64 prompt/off

And at least on my platform this was not a problem.

I can try to remove any strange character from file names but this will take some time.