Reduce File Size By Reducing Git History

[adrianklimczak]

Hi,

We have reached a limit of 2GB and we can't push any more to the remote. 

We have read the guidelines you provided.

However, we are not very happy with the approach of moving the head into the past, it would make more sense to move the tail forward, thus loosing the oldest history, which is no longer relevant to us anyway.  

Is it possible to reduce the size of the repository by shrinking the old git history?

[adrianklimczak · Accepted solution]

We wanted to keep the latest history of commits, so that we can keep the ability to review previous changes. Therefore we found a solution to remove the old history that contained some files that were later removed. Removing that history reduced the repository size from 2.04 GiB to 1.99 GiB. 

That was good but not enough, so we have to still move some more files to Git LFS, review the largest files and remove files that are no longer needed.

For those interested of how to delete the old history, we did the following:

git checkout --orphan temp e41d7f633c45c46bd42e97cecf93204191d9e4c9
git commit -m "Truncate history"
git rebase --onto temp e41d7f633c45c46bd42e97cecf93204191d9e4c9 master

temp - is the name of the branch that is going to be temporarily created for this operation.

After the temp is the SHA of the commit from, which we want to keep the history. Then we commit that branch. "Truncate history" is just a commit message.

Then we rebase our local master branch to that commit, which will become its first commit, everything before that commit will be lost. For more information refer to this website from, which we got this solution. https://passingcuriosity.com/2017/truncating-git-history/

This allowed us to get rid of any files that were deleted in the past but kept in the history, thus taking space.

After that we run to print all the files that are within the repository including all history.

git log --all --pretty=format: --name-only --diff-filter=D | sort -u

We found that there were still lots of files and folders that no longer exist in the repository but they are still taking space in the pack. By deleting the history of some of those files we managed to reduce the size of repository from 1.99 GiB to 1.61 GiB. We used the following command to do so:

git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch FILE_LIST' --prune-empty -f -- --all

We took this method from: https://blog.ostermiller.org/git-remove-from-history .

However, when doing git push --force it still fails with messages:

Uploading LFS objects: 100% (1556/1556), 288 MB | 331 KB/s, done
Counting objects: 15143, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8366/8366), done.
Writing objects: 100% (15143/15143), 116.28 MiB | 125.00 KiB/s, done.
Total 15143 (delta 7750), reused 13785 (delta 6763)
remote: Resolving deltas: 100% (7750/7750), done.
remote: Checking connectivity: 15143, done.
remote: Repository is in read only mode (over 2 GB size limit).
remote:
remote: Learn how to reduce your repository size: https://confluence.atlassian.com/x/xgMvEw.
To https://bitbucket.org/solidgamesstudio/three-defenders-2-ranger.git
! [remote rejected] develop -> develop (pre-receive hook declined)
error: failed to push some refs to 'https://adrianklimczak@bitbucket.org/solidgamesstudio/three-defenders-2-ranger.git'

Could you please explain why is that the case?

git count-objects -Hv
count: 0
size: 0 bytes
in-pack: 44188
packs: 1
size-pack: 1.61 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

[adrianklimczak]

We have fired a support ticket to resolve the issue of not being able to push the new repository size to the remote. Marking this as resolved, since we found the answers to the original question.

[Ana Retamal]

Hi Adrian, you won't be able to reduce the size of your repository like that. The purpose of resetting the head back is to remove the latest changes that have made your repo exceed the limit, not to re-write your history. If you'd like to rewrite your history that's also possible, although a more complex operation. My recommendation would be to run the following command to see how much space your repo is taking locally:

git count-objects -Hv

If you think this size can be reduced by deleting the history, you can go ahead and do so or, if you don't care about the history at all an easier alternative would be to just create a new repo with your current files, so the history is clean. 

For reference, here's our article on how to reduce the size of a repository.

Let us know if you have any questions!

Ana

[adrianklimczak]

Thank you for you answer. 

git count-objects -Hv

Prints "2 packs". Should there be 2 packs? Size is 2.04 GiB.