Unable to download file from Repository using Curl

[Krishna Kumar]

Hi,

We have a requirement to download a specific file from our repository to a path in EC2 instance. We following the second option listed in the below url which uses a CURL command to download a specific file from a repository.

https://confluence.atlassian.com/bbkb/how-to-download-specific-file-from-bitbucket-repository-1283492363.html

We are trying to download a tar.gz file. The command gets executed successfully but the file that is downloaded is of HTML type and not the actual file that we want. 

Can you please guide us to resolve the issue.

I have attached the snippet with the command used and the output.

Thanks

[Theodora Boudale]

Hi Krishna and welcome to the community!

I have tried reproducing this error by committing different tar.gz files in one of my repos and then trying to download them with an API call like yours, using curl. I haven't been able to reproduce this issue so far with different tar.gz files. Running the command file somefile.tar.gz correctly returns the type of the file as 'gzip compressed data'.

Can you please check and let me know the following:

1. If you use the credentials of a Bitbucket user for authentication in the API call, can you please ensure that you are using:

• the Bitbucket username of the user, and not the email address (the username is listed here https://bitbucket.org/account/settings/ if this is your own user account)
• an app password with at least Repositories - Read permissions (the account's password won't work, it can only be used for logging in to the website from browser)

2. The API call mentioned in the URL you posted is

curl -s -S --user <username>:<App Password> -L -O https://api.bitbucket.org/2.0/repositories/<WorkspaceID>/<RepoName>/src/<BranchName>/<FolderName>/<FileName>

<FolderName> should be the name of the folder where the file is located.

If the file is at the root of the repo, this will not be needed, so you need to use the following URL in the API call

https://api.bitbucket.org/2.0/repositories/<WorkspaceID>/<RepoName>/src/<BranchName>/<FileName>

If the file is in a subfolder of a folder in the repo, e.g. folder1/folder2/file.tar.gz then both folders should be included in the URL as follows

https://api.bitbucket.org/2.0/repositories/<WorkspaceID>/<RepoName>/src/<BranchName>/folder1/folder2/<FileName>

I see in the screenshot you posted here your URL looks like this:

https://api.bitbucket.org/2.0/repositories/<WorkspaceID>/<RepoName>/src/develop/coop/<FileName>

Does the file belong in a folder named coop that exists on the develop branch on your repo?

3. If you try to uncompress the tar.gz file you downloaded, does it uncompress properly and does it contain the compressed files?

4. If you clone the repo in that EC2 instance, checkout develop branch, pull the changes, and then run the same file command, does the output still show 'HTML document'?

Kind regards,
Theodora

[Krishna Kumar]

Thanks for the detailed response. Please see my comments below.

1. For authentication, username and app password(generated) were provided.

2. There is a folder named "coop" in the develop branch under which the file is present. As i had mentioned, the file provided in the path was found and downloaded but incomplete and as an HTML

3. No, we are unable to uncompress the file as it is of HTML type.

4. I cloned the repo in my EC2 and checked out the develop branch. The file is of type "gzip compressed data".

There are some other .gz files in the repo under different folders. I am able to successfully download them using the same command. The only difference I see now is with the size of the file. The .gz files i downloaded today are of 720 and 360 MB sized. The file with which we are facing the issue is around 1.4 GB. Could this be a cause for concern. If so, what can be done to download the file without any issues.

Thanks,

Krishna

[Theodora Boudale]

Hi Krishna,

Thank you for the info.

I believe the size of the file is the culprit here. I was able to reproduce this with a tar.gz file of a similar size. Our API requests will time out if they take more than 28 seconds, so the request times out before the whole file is downloaded. I'm afraid that the timeout is not configurable and we cannot increase it.

What you can do to get this file

1. You can download the source code of develop branch with a call like the following:

curl --request GET --url 'https://bitbucket.org/<WorkspaceID>/<RepoName>/get/develop.zip' -u <username>:<app_password> --output develop.zip

Since this is not using our API, the timeout will not apply. You can then unzip the file, copy-paste the tar.gz file into the location you want, and delete the rest.

This zip will contain all the source code of develop branch, without the commits and history.

2. Another option is to clone the develop branch of the repo with depth 1

git clone -b develop --depth 1 <bitbucket_repo>

Then, copy-paste the tar.gz file into the location you want and delete the clone.

Please feel free to let me know if these options work for you and if you have any questions.

Kind regards,
Theodora