How to store files on a bucket subdirectory using google-cloud-storage-deploy

[André Rugere]

Hi guys!

I built a simple pipeline to copy files from a folder and upload them to a GCS bucket using the google-cloud-storage-deploy pipe, as shown below:

# Copy dags files to GCS bucket
pipelines:
default:
- step:
name: 'Pushing files to GCS bucket' 
script:
- pipe: atlassian/google-cloud-storage-deploy:2.0.0
variables:
KEY_FILE: $GOOGLE_SERVICE_ACCOUNT_KEY
BUCKET: 'bucket-test'
SOURCE: 'files/'

However, I don't want these files to be stored in the root of bucket-test, I want them in a subdirectory called folder.

Is it possible to do that using the google-cloud-storage-deploy pipe? I couldn't find an example and got errors when trying to do that by passing bucket-test/folder as a value for the BUCKET variable.

So in summary, right now this pipeline is doing this:

.
├── folder/
├── file1
├── file2
└── file3

and I want it to do this:

.
└── folder/
    ├── file1
    ├── file2
    └── file3

Thanks for your help!

[Igor Stoyanov · Accepted solution]

Hi @André Rugere . We released new version 2.1.0 of the pipe, which allows you to upload to custom blob:

script:
  - pipe: atlassian/google-cloud-storage-deploy:2.1.0
    variables:
      KEY_FILE: $KEY_FILE
      BUCKET: 'my-bucket'
      SOURCE: 'myFile.txt'
      BLOB_NAME_PREFIX: 'my-folder/'

Regards, Igor

[André Rugere]

That's awesome, thanks for letting me know!

[Igor Stoyanov]

@André Rugere hi. Try to rename your files to `folderName/fileName` inside your `files/` folder.
Additional details: Google Cloud Storage simulated folders .

Regards, Igor

[André Rugere]

Hi @Igor Stoyanov , thanks for answering.

However, I don't think renaming our source files is a good approach in our case. In the documentation you sent me, there's an example using the gcloud command that aligns with the behavior I was expecting:

gcloud storage cp your-file gs : //your-bucket/abc/ 

As a result of this command, Cloud Storage creates an object named abc/your-file in the bucket your-bucket.

Notice that the bucket name includes /abc/, indicating the folder where the file should be placed.

I wish I could achieve the same with the google-cloud-storage-deploy pipe, for example:

BUCKET: 'bucket-test/folder'

Is it possible or is there any other similar way to do that?

[Igor Stoyanov]

@André Rugere , This feature is not supported in current implementation of the pipe.

We will investigate it and notify you when new pipe version with this feature will become available.

Regards, Igor

[André Rugere]

@Igor Stoyanov oh I see, no worries.

Are these pipes open to external contributors? I could try implementing this feature if you think it's reasonable. Let me know how I can help :) 

Thanks!

[Igor Stoyanov]

@André Rugere , pipes are open-sourced, you could suggest feature and follow release process: How to contribute .

Or you could wait until our team will implement this feature.

Regards, Igor