WARNING Repository size in Bitbucket is currently 3.9 GB

[marchinimarco]

Hi,

i have this warning since about 2 years and i'd like to solve it now.

My repository files are about 2GB but the warning is telling me that the repo is 3.9GB, so after digging deeper i found that my .git/objects folder in the repository is about 4.5GB. I think it's full of old references to old commits or old files no more in the repo or something similar.

I can't run the GC command by myself so i'm asking if some tech guy from Bitbucket can do it for me?

Thanks

This is the message i'm getting when i push a new commit:

remote: WARNING Repository size in Bitbucket is currently 3.9 GB

remote: Above the 4 GB size limit further commits are rejected

remote: Reduce repository size by removing large files from history

remote: HINT Learn how to reduce repository size: https://support.atlassian.com/bitbucket-cloud/docs/reduce-repository-size/#Rewind-history-to-undo-large-commits

[Theodora Boudale]

Hi and welcome to the community!

I ran a git gc on your repository but its size remains unchanged. The garbage collection only removes unreferenced commits (e.g. from deleted branches) and compresses file revisions. It does not remove commits that are part of the history or files that belong to the repo's history.

If you want to reduce the repo's size, you can check this reply of mine on another question for more details:

• https://community.atlassian.com/t5/Bitbucket-questions/Revert-bitbucket-repository-state-to-one-day-prior/qaq-p/1908360#M76762

If you decide to rewrite history to remove any large files and then push back to the repo, then the repo will need a git gc for the old references to get removed and for its size to get reduced. You can leave a reply here to let me know and I can run another git gc.

Please feel free to let me know if you have any questions.

Kind regards,
Theodora

[marchinimarco]

Hi Theodora,

thanks a lot for your help here.

I followed your detailed steps from the other thread and i see the following entries as the biggest files in the repository. All rows belongs to the same file, it's a binary geolocation IP addresses database that i need to deploy in production.

There are multiple entries because every N months i overwrite the same file with a new version of it with fresh IPs and then i commit to the repository in order to distribute the updated file to the production servers, but it's one file only in the repository. I think this is the cause of the huge repo size.

What should i do now? Should i remove it the file from tracking? but after how can i deploy it with the code to the server?

Thank you

[Theodora Boudale]

Hi,

If you keep the file in the repo and then keep committing different versions of it, the repo's size will eventually go over the 4 GB limit and then the repo will become read-only.

If you need to version this file, I suggest considering Git LFS (track this file with Git LFS):

• https://support.atlassian.com/bitbucket-cloud/docs/manage-large-files-with-git-large-file-storage-lfs/
• https://support.atlassian.com/bitbucket-cloud/docs/use-git-lfs-with-existing-bitbucket-repositories/

Git LFS files are stored outside the core Git repo on different servers and the Git repo only contains pointers to the LFS files.

Workspaces on the Free billing plan have 1 GB of Git LFS in total (for all repos in that workspace), but if you change your billing plan to Free with Overages, you'll get an additional 100 GB of Git LFS storage for $10/month. If you then use over 100 GB, an additional 100 GB of Git LFS storage will be added to your plan for a total of $20/month etc.

If you don't need to version this file, you can look into file storage services for storing this file externally and then use curl or wget on the production servers to get this file. Bitbucket Cloud repos also have a Downloads section where you can upload files (and you can then download files with curl), but the limit is 1 GB and your file is close to this limit, so the operation may time out. If that's the case, an external file storage solution may be best.

You could then rewrite history in the repo to remove this file from history, so that the repo contains only your source code.

Kind regards,
Theodora

[marchinimarco]

Hi Theodora,

thank you very much for your support!

I've just removed the incriminated file using BFG following the point 2.a from your guide step by step. Checking the repo size using the command 

git count-objects -Hv

it effectively changed from 3.88GB to 1.07GB. Look like everything is good so i've pushed the change from the local mirrored clone to the remote repository on Bitbucket. But i'm getting the following error on git push:

remote: Your push has been blocked because it includes at least one file that is 100 MB or larger. Bitbucket Cloud recommends using Git LFS for large files to prevent repositories from growing too large and reaching the 4 GB repo size limit. Learn more about Git LFS at https://support.atlassian.com/bitbucket-cloud/docs/manage-large-files-with-git-large-file-storage-lfs/. If necessary, the file size limit can be disabled by an admin in Repository settings. Learn more at https://support.atlassian.com/bitbucket-cloud/docs/reduce-repository-size/#Avoid-adding-large-files.

To bitbucket.org:my_repo.git

! [remote rejected]  master -> master (pre-receive hook declined)

error: failed to push some refs to 'bitbucket.org:my_repo.git'

Am i doing something wrong?

Thank you

[Theodora Boudale]

Hi,

Do you have any other files on this repo that are larger than 100 MB?

We block pushes that have files over 100 MB, but this can be disabled from the Repository settings of the repo on our website > Repository details. On this page, if you click ADVANCED you will see an option Block pushes with files over 100MB that you can uncheck.

I also double-checked again BFG's page here https://rtyley.github.io/bfg-repo-cleaner/, and I see that by default the BFG doesn't modify the contents of your latest commit on your master (or 'HEAD') branch. So, if that large database file is also on the last commit, BFG may not have removed the file from that commit.

As per BFG's documentation (section "Your current files are sacred"), you can use the --no-blob-protection flag to remove the file from the latest commit as well.

You can then push to a newly created empty Bitbucket repo first to confirm if the file is removed and if the repo looks ok, before pushing to your current repo.

Please keep in mind that your current repo's size will probably go up after you push the changes you made with BFG, because the repo will contain both old and new references. It will need a git gc so that the old references get removed, so you can leave a comment here and I will run it.

Kind regards,
Theodora

[marchinimarco]

Hi,

yes we have other files >100mb indeed. I'll redo all the steps including your last suggestions regarding the latest commit and get back to you once finished.

Thank you

Marco

[Theodora Boudale]

Hi Marco,

Sure, I will wait for your update.

Kind regards,
Theodora

[marchinimarco]

Hi Theodora,

i've just redo all the steps, cleaned all big files with BFG and pushed the changes to the remote repository, now i see a size of 557Mb. 

The original repository is still 3.89Gb so i suppose it's time to execute your GC command right? After that i have to delete my old local copy and start with a new clone right?

Thank you

Below the result of the git push:

Writing objects: 100% (50099/50099), 557.56 MiB | 2.14 MiB/s, done.

Total 50099 (delta 0), reused 0 (delta 0), pack-reused 50099

remote: Resolving deltas: 100% (33157/33157), done.

remote: WARNING: Your push includes at least one file that is 50 MB or more. Bitbucket Cloud recommends using Git LFS for large files to prevent repositories from growing too large and reaching the 4 GB repo size limit. Learn more about Git LFS at https://support.atlassian.com/bitbucket-cloud/docs/manage-large-files-with-git-large-file-storage-lfs/. If your repository reaches the 4 GB size limit, all pushes will be blocked. Learn more about the repository size limit at https://support.atlassian.com/bitbucket-cloud/docs/reduce-repository-size/.WARNING Repository size in Bitbucket is currently 3.9 GB

[Theodora Boudale]

Hi Marco,

Thank you for the update.

I ran another git gc on the repo, and its size has now been reduced to 565.6 MB.

Does it look ok on your end?

You may want to keep your old copy for backup purposes but yes, start with a new clone for any new commits from now on. Otherwise, you will push the old commits with the large file back to the remote repo and you'll need to clean it up again.

Kind regards,
Theodora

[marchinimarco]

Wow Theodora,

everything is perfect now, thanks a lot for your support!!

Best, Marco

[Theodora Boudale]

That's good to hear and you are very welcome, I'm glad to have helped!

Please feel free to reach out if you ever need anything else!

[marchinimarco]

Anyone from the tech team? 

Thank you