nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Using google deploy pipe as an option to publish artifact larger than 1 GB in bitbucket pipeline

yogesh_devi

Hi Pipeline Gurus,

I have a bitbucket pipeline that generates an artifact that is greater than 1GB

Since when originally the arifact was created I saw message as follows

2 Artifact pattern <span>deploy</span>/ubuntu/pkgs/*.deb matched 1 files with a total size of 1.2 GiB
3 Compressed files for artifact "<span>deploy</span>/ubuntu/pkgs/*.deb" matching pattern <span>deploy</span>/ubuntu/pkgs/*.deb to 1.2 GiB in 59 seconds<br>4 Compressed artifact size is 251.7 MiB over the 1 GiB upload limit, so the artifact will not be uploaded.<span>1 Searching for files matching artifact pattern deploy/ubuntu/pkgs/*.deb<br></span>

After some reading I figured there is no presently supported direct mechanism to store and download an artifact greater than 1 Gigabyte ; I will be grateful if some tells me what are my options to handle this size artifact ( 1.26 GB)

I looked at https://support.atlassian.com/bitbucket-cloud/docs/deploy-to-google-cloud/ and based on that I am exploring option of deploying that artifact directly to a GCS bucket . The pipeline definition is as follows.

image: atlassian/default-image:3

options:
size: 2x
max-time: 90

pipelines:
  branches:
    mybranches*:
    - step: 
      name: 'invoke the build commands'
      script:
        - echo "invoking the necessary build commands that run OK "
        - echo "build commands generate artifact 'deploy/ubuntu/pkgs/mypackage_amd64.deb'"
        - pipe: atlassian/google-app-engine-deploy:1.7.0
          variables:
            KEY_FILE: $KEY_FILE
            PROJECT: 'my-google-project'
            DEPLOYABLES: deploy/ubuntu/pkgs/*.deb
            VERSION: '${BITBUCKET_BUILD_NUMBER}'
            BUCKET: 'gs://bucket-for-deploying-artifacts'
            PROMOTE: 'true'
            STOP_PREVIOUS_VERSION: 'true'
            EXTRA_ARGS: '--verbosity=debug'
      artifacts: # defining the artifact for download 
        - deploy/ubuntu/pkgs/*.deb

However when run this pipeline I get an error as follows

<span>Status: Downloaded newer image for bitbucketpipelines/google-app-engine-deploy:1.7.0</span>

90 INFO: Setting up environment.
91 echo "${KEY_FILE}" | base64 -d >> /tmp/key-file.json
92 gcloud auth activate-service-account --key-file /tmp/key-file.json --quiet --verbosity=warning
93 Activated service account credentials for: [myaccount@mygoogleproject.iam.gserviceaccount.com]
94 gcloud config set project mygoogleproject --quiet --verbosity=warning
95 Updated property [core/project].
96 INFO: Starting deployment to GCP app engine...
97 gcloud app --quiet deploy deploy/ubuntu/pkgs/mypackage_amd64.deb --version=16 --bucket gs://bucket-for-deploying-artifacts --promote --stop-previous-version --verbosity=debug --verbosity=warning
98 ERROR: (gcloud.app.deploy) [/opt/atlassian/pipelines/agent/build/deploy/ubuntu/pkgs/mypackage_amd64.deb] could not be identified as a valid source directory or file.
99 Deployment failed.

Please note that some details like project, package and bucket names are synonymized and pardon if any inconsistencies have crept in the above description owing to that.

Sincere thanks for your response

Yogesh

Accepted answers

John Emmanuel

Hi,

You don’t need to declare artifacts in the step unless you specifically want to download them from the Bitbucket Pipelines UI or pass them to subsequent steps.

From your setup, it looks like you’re generating a .deb file using some commands and then deploying it to a Google Cloud bucket using a pipe.

The error you’re encountering likely indicates a mismatch in the file path you're providing to the pipe. Make sure you're in the correct directory (cd to it if necessary) and use the appropriate relative path to the file or folder. Alternatively, check whether the pipe supports absolute paths and verify if the pattern you're using—deploy/ubuntu/pkgs/*.deb—is valid. If the pipe doesn't support file globs or specific files, consider pointing it to the folder instead.

In either case, there's no need to declare the .deb files as artifacts in your Bitbucket pipeline configuration.

John

All comments

John Emmanuel

Hi,

You don’t need to declare artifacts in the step unless you specifically want to download them from the Bitbucket Pipelines UI or pass them to subsequent steps.

From your setup, it looks like you’re generating a .deb file using some commands and then deploying it to a Google Cloud bucket using a pipe.

In either case, there's no need to declare the .deb files as artifacts in your Bitbucket pipeline configuration.

John

yogesh_devi

Hi John - I am specifying path that is valid in the step

The base directory of step - where the git checkout happens

the artifact is generated in relative path deploy/ubuntu/pkgs

. The error that I cited earlier viz.

2 Artifact pattern <span>deploy</span>/ubuntu/pkgs/*.deb matched 1 files with a total size of 1.2 GiB
3 Compressed files for artifact "<span>deploy</span>/ubuntu/pkgs/*.deb" matching pattern <span>deploy</span>/ubuntu/pkgs/*.deb to 1.2 GiB in 59 seconds<br>4 Compressed artifact size is 251.7 MiB over the 1 GiB upload limit, so the artifact will not be uploaded.<span>1 Searching for files matching artifact pattern deploy/ubuntu/pkgs/*.deb<br></span>

The above error appears when I declare an artifact with the path

        - deploy/ubuntu/pkgs/*.deb

So the artifcat is around.

not just that

I introduced a debug ls in my pipeline

ls -l deploy/ubuntu/pkgs/*.deb

ls -l deploy/ubuntu/pkgs/*.deb

-rw-r--r-- 1 root root 1337441016 Apr 30 12:33

deploy/ubuntu/pkgs/mypackage_amd64.deb

so the artifact is pretty much there is pipeline step in the path deploy/ubuntu/pkgs/mypackage_amd64.deb

Maybe I need to understand what path does pipe start with - is it the directory in which the calling step was ? Or is it something else ?

Or is it that pipe does not expand wildcards ; does it need a specific artifact name ?

John Emmanuel

This error:

2 Artifact pattern <span>deploy</span>/ubuntu/pkgs/*.deb matched 1 files with a total size of 1.2 GiB
3 Compressed files for artifact "<span>deploy</span>/ubuntu/pkgs/*.deb" matching pattern <span>deploy</span>/ubuntu/pkgs/*.deb to 1.2 GiB in 59 seconds<br>4 Compressed artifact size is 251.7 MiB over the 1 GiB upload limit, so the artifact will not be uploaded.<span>1 Searching for files matching artifact pattern deploy/ubuntu/pkgs/*.deb<br></span>

This is due to Bitbucket's limitation of allowing only up to 1GB of artifacts. So your artifacts cannot be declared as a Bitbucket pipeline artifact. You may need to use something like S3 or Google Cloud Storage to store your artifacts.

For your use case, I don't see any need to declare them as artifacts. You can leave it as is or move to a Google Cloud bucket for later reference.

Ref: https://support.atlassian.com/bitbucket-cloud/docs/use-artifacts-in-steps/

For the error I was talking about:

98 ERROR: (gcloud.app.deploy) [/opt/atlassian/pipelines/agent/build/deploy/ubuntu/pkgs/mypackage_amd64.deb] could not be identified as a valid source directory or file.
99 Deployment failed.

From the pipe's readme, I could find that you can use a whitespace-separated list of files. It may not support the glob patterns (*.deb, etc). If you are unsure, try pointing out the entire directory.

Ex:

DEPLOYABLES: 'file1 file2'

In your case:

DEPLOYABLES: deploy/ubuntu/pkgs/

Ref: https://bitbucket.org/atlassian/google-app-engine-deploy/src/master/

John Emmanuel

For question 3, Bitbucket Pipes usually start from the repository root — also known as the Bitbucket clone directory, which can be referenced by the $BITBUCKET_CLONE_DIR environment variable. You should reference files and folders relative to this directory, or use the variable to construct full paths.

This behavior holds unless the pipe explicitly changes its working directory internally — which, based on the documentation, your current pipe does not appear to do.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

cloud

bitbucket

bitbucket-cloud

bitbucket-pipelines

pipes

gcp