Repo on BitBucket over limit, local has been cleaned of large files, how can I push?

[Ricardo Pereira]

Hello

I have accidentally uploaded a folder with a large number of files and we have exceeded the allowed limit of 4Gb.

I have deleted the files locally, but I can no longer commit because I always get the same error. I have executed different commands that I have found in this forum, but it seems that none of them work for me,

I think the only thing I need is to run the garbage collector. Can you please help me?

Thanks

[Theodora Boudale]

Hi @Ricardo Pereira,

I can see in our system the repo over the 4 GB limit from the ones that you have access to. I ran a git gc for this repo, but its size remains unchanged.

Could you please let me know what changes you've made to your local repo?

• Did you do a git reset to undo the large commit? In this case, you should still be able to push (a git reset of the problematic commit can be pushed even if the repo is over the 4 GB limit)
• Did you commit file deletions?
• Or did you rewrite history with a tool like BFG or git filter-branch?

In the last 2 cases you won't be able to push, the only thing that can be pushed to a repo over the 4 GB limit is a git reset and no new commits.

Please let me know which of the 3 changes you have made and what is the output of git count-objects -Hv in your local repo so I can suggest further actions.

Kind regards,
Theodora

[Ricardo Pereira]

Hello Theodora,

thanks for the quick answer.

After the   git count-objects -Hv this is the result.

count: 947
size: 5.15 GiB
in-pack: 0
packs: 0
size-pack: 0 bytes
prune-packable: 0
garbage: 8
size-garbage: 1.91 GiB

I have tried to delete files and use the git reset, but did no worked.

[Ricardo Pereira]

The problem is that on my last commit I forgot to send a Git.ignore file. 

I hope this helps with my problem. Thanks.

[Theodora Boudale]

Hi Ricardo,

The output of git count-objects -Hv shows that your local repo has a size of 5.15 GiB.

If you inspect the history of your local repo:

• is the problematic commit (with the large files) still part of the repo's history?
• if so, have you merged the branch with this commit to any other branch?

If the answer to both questions is no, you can try running a garbage collection on your local repo with the command

git reflog expire --expire=now --all && git gc --prune=now --aggressive

and then inspect the size again git count-objects -Hv

If the problematic commit is still part of the history, and you have not merged the branch to any other branch, you can check the steps mentioned in the link below on how to undo the last push:

• https://support.atlassian.com/bitbucket-cloud/docs/reduce-repository-size/#Undoing-the-last-push

If the answer to both questions I asked before is yes, or if the options above do not work, I would suggest trying to remove the folder with the large files from the repo's history with BFG:

• https://rtyley.github.io/bfg-repo-cleaner/

This page provides step-by-step instructions on how to do that. You can use BFG with the argument --delete-folders (mentioned in the link) in order to remove it from the repo's history.

Some important notes about BFG and rewriting history:

• Always take a backup of the repo before you proceed with any of the changes I suggested above, in case you accidentally delete something you didn't want to and want to recover the repo to the prior state. You can take a backup by cloning with the --mirror flag (other than the one you will use for BFG):

git clone --mirror repo_url

• BFG does a history rewrite. This means that the commit hashes of the repository will change. It's good to communicate this to your team so there are no surprises, and also so that they take a fresh clone of the repo after you push your changes (and avoid pushing the old history back)
• After you make changes with BFG, I would suggest pushing them to a newly created repo first, in order to inspect the history and confirm if it looks ok.

When you have a local repo of a smaller size (you can check that with the command git count-objects -Hv, specifically the fields size and size-pack in the output), please let me know, and I can temporarily increase the repo size limit so you can push your changes.

If you have any questions, please feel free to let me know.

Kind regards,
Theodora

[Ricardo Pereira]

Hello Theodora.

Once again thank you for the help.

I am truing the bfg option, but 'm not able to  repair the repo.

IT is not possibel to have for a few moment, like 20 minutes the chane to make a push ??

Because i removed the large file localy already.

 git count-objects -Hv
count: 92
size: 292.92 KiB
in-pack: 840
packs: 1
size-pack: 2.71 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

[Theodora Boudale]

Hi Ricardo,

The current output of git count-objects -Hv shows a repo of a much smaller size, so of course, I have temporarily raised the size limit for this repo to 6 GB so you can push.

Please note that the repo's size in Bitbucket may not get reduced immediately after you push, another garbage collection may be needed after you push your changes.

Please feel free to let me know when you push the changes, and I can run another git gc for the repo.

Kind regards,
Theodora

[Ricardo Pereira]

Hi Teodora, i have made the push with the changes. 

Once again, thank you for the belo.

Best wishes.

[Theodora Boudale]

Hi Ricardo,

I ran another git gc for the repo, but its size remains unchanged.

1. Could you please let me know what changes you made to reduce the size?

2. If you open the Commits page of the repo on Bitbucket website, do you see there the commit with the large files?

3. Could you take a mirror clone with the repo, with the command

git clone --mirror repo_url

and then navigate inside the directory of the mirror clone, execute git count-objects -Hv and let me know what output you get?

I haven't changed the repo size limit back to 4 GB yet until we figure out the culprit.

Kind regards,
Theodora

[Ricardo Pereira]

Hello Theodora, 

 i haved removed the .vagrant files from the repo, and put the indication in the git.ignore file.

I even removed everything from the repo but somehow ,the repo is still over the  4gb capacity.

2. If you open the Commits page of the repo on Bitbucket website, do you see there the commit with the large files?

Yes is the commit 68a7c17. 

I think i found the culprit is a PACK file, in the .git/objects/pack path. 

(pack-24eea3a06b56d08a49a2679ed3ee115aa13a2e17.pack).

There is a way to remove&resolve hte file _

Once again thanks for the help and patience .

Best wishes.

[Theodora Boudale]

Hi Ricardo,

The pack file contains all objects of a git repo, if you remove it there will be no repo.
If that file is large it's usually because large files still exist in history.

If you have committed the deletion of large files (I'm not sure if this is what you have done), these large files will still remain part of the history in the older commits.

If you open commit 68a7c17 on Bitbucket website, are the large files listed there?

Kind regards,
Theodora

[Ricardo Pereira]

Hi Theodora,

If you open commit 68a7c17 on Bitbucket website, are the large files listed there?

Yes they are.

[Theodora Boudale]

Hi Ricardo,

Thank you for your reply.

It seems that this is the reason that the repo is still large. If the files are listed in that commit, they are still part of the Git history, even if you commit their deletion later on.

I would suggest trying BFG, which is a tool that can remove files from a repo's history:

• https://rtyley.github.io/bfg-repo-cleaner/

I am listing below step by step instructions:

1. Take a mirror clone of the repo

git clone --mirror repo-url

2. Copy and paste the directory of the mirror clone elsewhere and keep that as a backup.

3. Download the BFG jar file from the link above, and put it in the same directory as your mirror clone directory from step 1.

4. You mentioned in your first post that you uploaded a folder with large files.
If you want to remove the whole folder from the repo history, you can use the option --delete-folders with BFG

In a terminal application, navigate to the directory where the mirror clone AND bfg-1.14.0.jar are located, and execute the following:

bfg-1.14.0.jar --delete-folders myfolder --no-blob-protection my-repo.git

Replace myfolder with the name of the folder with the large files
Replace my-repo.git with the name of the mirror clone directory from step 1

5. When the process above finishes, navigate to the directory of the mirror clone and perform a garbage collection

cd my-repo.git
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive

Replace my-repo.git with the name of the mirror clone directory

Afterwards, check the size with git count-objects -Hv

If the size is reduced, then the removal was most likely successful.

6. Create a new empty Bitbucket Cloud repo

7. Push from that mirror clone to the newly created empty Bitbucket repo first, in order to inspect if the changed repo is in good shape.

8. If the repo in step 7 looks ok, you can push to the existing repo and let me know so I can run a garbage collection.

As I mentioned earlier, if you use BFG, the commit hashes of the repository will change. It's good to communicate this to your team so there are no surprises, and also so that they take a fresh clone of the repo after you push your changes and avoid pushing the old history back.

If you have any questions, please feel free to let me know.

Kind regards,
Theodora

[Ricardo Pereira]

Hi Theodora, 

I think i finnaly resolved the problem, using the bfg tool.

Once again thanks for help and patience.

Best wishes.

[Theodora Boudale]

Hi Ricardo,

That's good to hear and you are very welcome, I'm glad to have helped!

I ran another git gc for your repo, and its size is 3.5 MB now. Can you see the updated size for the repo?

If other users are working on this repo as well and they have pulled the large commit locally, I would suggest asking them to take a new clone in order to avoid pushing back that commit.

I have changed the repo size limit to 4 GB as well since the issue is resolved.

Please feel free to let me know if you need anything further.

Kind regards,
Theodora

[Ricardo Pereira]

Hi Theodora,

I ran another git gc for your repo, and its size is 3.5 MB now. Can you see the updated size for the repo?

I can confirm the size is 3.5MB.

Once again thanks.

Best Wishes.

[Theodora Boudale]

You are very welcome Ricardo, please feel free to reach out if you ever need anything else!

Kind regards,
Theodora