Using google deploy pipe as an option to publish artifact larger than 1 GB in bitbucket pipeline

[yogesh_devi]

Hi Pipeline Gurus,

I have a bitbucket pipeline that generates an artifact that is greater than 1GB 

Since when originally the arifact was created I saw message as follows 

1 Searching for files matching artifact pattern deploy/ubuntu/pkgs/*.deb
2 Artifact pattern deploy/ubuntu/pkgs/*.deb matched 1 files with a total size of 1.2 GiB
3 Compressed files for artifact "deploy/ubuntu/pkgs/*.deb" matching pattern deploy/ubuntu/pkgs/*.deb to 1.2 GiB in 59 seconds
4 Compressed artifact size is 251.7 MiB over the 1 GiB upload limit, so the artifact will not be uploaded.

After some reading I figured there is no presently supported direct mechanism to store and download an artifact greater than 1 Gigabyte ; I will be grateful if some tells me what are my options to handle this size artifact ( 1.26 GB) 

I looked at https://support.atlassian.com/bitbucket-cloud/docs/deploy-to-google-cloud/ and based on that I am exploring option of deploying that artifact directly to a GCS bucket  . The pipeline definition  is as follows. 

image: atlassian/default-image:3

options:
size: 2x
max-time: 90

pipelines:
  branches:
    mybranches*:
    - step: 
      name: 'invoke the build commands'
      script:
        - echo "invoking the necessary build commands that run OK "
        - echo "build commands generate artifact 'deploy/ubuntu/pkgs/mypackage_amd64.deb'"
        - pipe: atlassian/google-app-engine-deploy:1.7.0
          variables:
            KEY_FILE: $KEY_FILE
            PROJECT: 'my-google-project'
            DEPLOYABLES: deploy/ubuntu/pkgs/*.deb
            VERSION: '${BITBUCKET_BUILD_NUMBER}'
            BUCKET: 'gs://bucket-for-deploying-artifacts'
            PROMOTE: 'true'
            STOP_PREVIOUS_VERSION: 'true'
            EXTRA_ARGS: '--verbosity=debug'
      artifacts: # defining the artifact for download 
        - deploy/ubuntu/pkgs/*.deb

 However when run this pipeline I get an error as follows 

Status: Downloaded newer image for bitbucketpipelines/google-app-engine-deploy:1.7.0

90 INFO: Setting up environment.
91 echo "${KEY_FILE}" | base64 -d >> /tmp/key-file.json
92 gcloud auth activate-service-account --key-file /tmp/key-file.json --quiet --verbosity=warning
93 Activated service account credentials for: [myaccount@mygoogleproject.iam.gserviceaccount.com]
94 gcloud config set project mygoogleproject --quiet --verbosity=warning
95 Updated property [core/project].
96 INFO: Starting deployment to GCP app engine...
97 gcloud app --quiet deploy deploy/ubuntu/pkgs/mypackage_amd64.deb --version=16 --bucket gs://bucket-for-deploying-artifacts --promote --stop-previous-version --verbosity=debug --verbosity=warning
98 ERROR: (gcloud.app.deploy) [/opt/atlassian/pipelines/agent/build/deploy/ubuntu/pkgs/mypackage_amd64.deb] could not be identified as a valid source directory or file.
99 Deployment failed.

Please note that some details like project, package and bucket names are synonymized and pardon if any inconsistencies have crept in the above description owing to that.  

Sincere thanks for your response

Yogesh 

[John Emmanuel · Accepted solution]

Hi,

You don’t need to declare artifacts in the step unless you specifically want to download them from the Bitbucket Pipelines UI or pass them to subsequent steps.

From your setup, it looks like you’re generating a .deb file using some commands and then deploying it to a Google Cloud bucket using a pipe.

The error you’re encountering likely indicates a mismatch in the file path you're providing to the pipe. Make sure you're in the correct directory (cd to it if necessary) and use the appropriate relative path to the file or folder. Alternatively, check whether the pipe supports absolute paths and verify if the pattern you're using—deploy/ubuntu/pkgs/*.deb—is valid. If the pipe doesn't support file globs or specific files, consider pointing it to the folder instead.

In either case, there's no need to declare the .deb files as artifacts in your Bitbucket pipeline configuration.

~

John :)

[yogesh_devi]

Hi John - I am  specifying path that is valid in the step

The base directory of step - where the git checkout happens 

the artifact is generated in relative path deploy/ubuntu/pkgs 

.  The error that I cited earlier viz.

1 Searching for files matching artifact pattern deploy/ubuntu/pkgs/*.deb
2 Artifact pattern deploy/ubuntu/pkgs/*.deb matched 1 files with a total size of 1.2 GiB
3 Compressed files for artifact "deploy/ubuntu/pkgs/*.deb" matching pattern deploy/ubuntu/pkgs/*.deb to 1.2 GiB in 59 seconds
4 Compressed artifact size is 251.7 MiB over the 1 GiB upload limit, so the artifact will not be uploaded.

The above error appears when I declare an artifact with the path 

        - deploy/ubuntu/pkgs/*.deb

 So the artifcat is around.

not just that 

I introduced a debug ls in my pipeline 

ls -l deploy/ubuntu/pkgs/*.deb

ls -l deploy/ubuntu/pkgs/*.deb

-rw-r--r-- 1 root root 1337441016 Apr 30 12:33

deploy/ubuntu/pkgs/mypackage_amd64.deb

so the artifact is pretty much there is pipeline step in the path deploy/ubuntu/pkgs/mypackage_amd64.deb

Maybe I need to understand what path does pipe start with  - is it the directory in which the calling step was ? Or is it something else ? 

Or is it that pipe does not expand wildcards ; does it need a specific artifact name ?

[John Emmanuel]

I

This error:

1 Searching for files matching artifact pattern deploy/ubuntu/pkgs/*.deb
2 Artifact pattern deploy/ubuntu/pkgs/*.deb matched 1 files with a total size of 1.2 GiB
3 Compressed files for artifact "deploy/ubuntu/pkgs/*.deb" matching pattern deploy/ubuntu/pkgs/*.deb to 1.2 GiB in 59 seconds
4 Compressed artifact size is 251.7 MiB over the 1 GiB upload limit, so the artifact will not be uploaded.

This is due to Bitbucket's limitation of allowing only up to 1GB of artifacts. So your artifacts cannot be declared as a Bitbucket pipeline artifact. You may need to use something like S3 or Google Cloud Storage to store your artifacts.

For your use case, I don't see any need to declare them as artifacts. You can leave it as is or move to a Google Cloud bucket for later reference.

Ref: https://support.atlassian.com/bitbucket-cloud/docs/use-artifacts-in-steps/

II

For the error I was talking about:

98 ERROR: (gcloud.app.deploy) [/opt/atlassian/pipelines/agent/build/deploy/ubuntu/pkgs/mypackage_amd64.deb] could not be identified as a valid source directory or file.
99 Deployment failed.

From the pipe's readme, I could find that you can use a whitespace-separated list of files. It may not support the glob patterns (*.deb, etc). If you are unsure, try pointing out the entire directory.

Ex:

DEPLOYABLES: 'file1 file2'

In your case: 

DEPLOYABLES: deploy/ubuntu/pkgs/

Ref: https://bitbucket.org/atlassian/google-app-engine-deploy/src/master/

[John Emmanuel]

For question 3, Bitbucket Pipes usually start from the repository root — also known as the Bitbucket clone directory, which can be referenced by the $BITBUCKET_CLONE_DIR environment variable. You should reference files and folders relative to this directory, or use the variable to construct full paths. 

This behavior holds unless the pipe explicitly changes its working directory internally — which, based on the documentation, your current pipe does not appear to do.

[yogesh_devi]

Hi John -

A huge thanks for your response.

Couple things - I moved to a newer version of pipe viz: " atlassian/google-cloud-storage-deploy:2.2.0" - not owing to any specific reason but just so that I use a more late version of pipe that hopefully is better.

And yes- Removing the wildcard from filename and specifying file name specifically allowed me to advance further. Now I no longer get the error "could not be identified as a valid source directory or file."   

Now I am stuck at next error ; the messages leading to that error are as follows 

pipe: atlassian/google-cloud-storage-deploy:2.2.0
...
INFO: Using default authentication with KEY_FILE.
INFO: Starting deployment to GCP storage...
...
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py"
 line 2697, in _prep_and_do_upload created_json = self._do_upload(
...
File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 4905, in _raise_from_invalid_response    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.NotFound: 404 POST https://storage.googleapis.com/upload/storage/v1/b/gs://mybucket/o?uploadType=resumable: Not Found: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.CREATED: 201>)

I tried to validate following things and found ok

- The gs://mybucket does exist in  'my-google-project'

-that the key  ( base 64 encoded key associated with the vairable KEY_FILE) belongs to a service account and that account has following permissions on gs://mybucket

 - storage admin

- storage object admin

- storage legacy bucket owner

I am kind of stuck - your comments welcome 

If I have a breakthrough I will post here 

Thanks much !

Yogesh 

[yogesh_devi]

Finally I figured the problem 

Looks like in version 2.2.0 pipe that i use "

pipe: atlassian/google-cloud-storage-deploy:2.2.0

The value against BUCKET field DOES NOT EXPECT the gs:// PREFIX in version 2.2.0 pipe 

so this one little change did the trick for me 

 BUCKET: 'mybucket'

Sadly the examples I referred to were for older version  ( 1.7.0)  of this pipe and they showed the bucket name with a gs:// prefix  ( like https://support.atlassian.com/bitbucket-cloud/docs/deploy-to-google-cloud/ ) . Can atlassian support please fix those examples so they reflect the behaviour of contemporary version of this pipe  

and John,   since your response  helped me move ahead - the answer does deserve to be accepted So I accept it with gratitude ; thank you 

[John Emmanuel]

@yogesh_devi Thanks for replying with the actual solution so that later people can benefit from it :)

Also you can contribute to this pipe if you have some spare time. Maybe you can update the readme file with what you have found with a PR.

https://bitbucket.org/atlassian/google-app-engine-deploy/src/master/CONTRIBUTING.md

Addtionally, have you explored connecting your pipelines with your GCP using OIDC? its a better way that eliminates the need to use access keys / service account keys.

https://support.atlassian.com/bitbucket-cloud/docs/integrate-pipelines-with-resource-servers-using-oidc/

https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds

Cheers :)