Repo size not reduced after using BFG to move binary files to LFS

[Alan Cordner]

I have followed the instructions on the following page in an attempt to rewrite the history of my 1.9GB repo, which is largely comprised of binary files, to use LFS.

https://confluence.atlassian.com/bitbucket/use-bfg-to-migrate-a-repo-to-git-lfs-834233484.html

The process completes successfully, and I can see the .gitattributes file has been added to a very early commit in the repo, and it includes file types for all of the binary file types in the repo.

However, in Bitbucket, the size of the repo is still reported as 1.9GB with 0 files in LFS. I was expecting to see the repo size shrink to almost nothing and the binary files to appear in LFS. I cannot figure out what I am doing wrong.

BFG version: 1.13.0

Any assistance would be great!

[Daniil Penkin · Accepted solution]

Hello @Alan Cordner ,

The size of a repository in Bitbucket after such cleanup might not update immediately because garbage collection isn't triggered after every push. This is due to the fact GC is potentially quite an expensive operation, especially on large repositories.

The repository I think you were describing has reduced in size drastically, however it indeed has no files in LFS which means that you haven't uploaded converted large files to LFS. Not sure which step you missed, but to fix that now you need to run git lfs push in your converted repository.

Hope this helps.

Cheers,
Daniil

[Alan Cordner]

Daniil,

Thank you for the reply. If git lfs push is required, why does the article I linked above not call it out? It seems like an incomplete process. I waited overnight, and the repo still has not changed in size, and LFS files still indicate 0. 

Using SourceTree on a converted repo, am I now going to have to do a push AND a lfs push when committing new or updated binary files? Or does SourceTree recognize LFS is being used and automatically does both?

[Daniil Penkin]

It shouldn't be required if you installed LFS client correctly (which means that LFS hooks are installed). This might be not straightforward if you have using any kind of Git hooks manager.

git lfs push is an explicit way to tell LFS client to upload tracked files.

I waited overnight, and the repo still has not changed in size, and LFS files still indicate 0.

So what is the repository in question? The lag with reflecting size correctly only happens when the repository shrinks. Size of files in LFS is always accurate, so if it is indicating 0, then no files have been uploaded to LFS.

If you have LFS client installed correctly, it is enough to run just git push, the pre-push hook installed by LFS client should trigger pushing of tracked large files to LFS. AFAIK, SourceTree doesn't add anything special to the LFS setup, it has to rely on Git configuration. Again, running both operations shouldn't be required if LFS client was initialised.

Hope this makes sense.

Cheers,
Daniil

[Alan Cordner]

Daniil,

I was able to repeat the process, and this time it worked as expected. There seem to be a few minor details left out of the instructions on the page referenced, and not being a Git expert they threw me off, but I was able to work through them.

I had to repeat a few operations multiple times due to fatal errors. For example:

acordner@PC-AMF-507CL13 MINGW32 /c/temp
$ git clone --mirror https://acordner@bitbucket.org/xxxxxx/devices.git
Cloning into bare repository 'devices.git'...
remote: Counting objects: 1849, done.
fatal: the remote end hung up unexpectedly
fatal: protocol error: bad pack header

acordner@PC-AMF-507CL13 MINGW32 /c/temp
$ git clone --mirror https://acordner@bitbucket.org/xxxxxx/devices.git
Cloning into bare repository 'devices.git'...
remote: Counting objects: 1849, done.
fatal: the remote end hung up unexpectedly
fatal: protocol error: bad pack header

acordner@PC-AMF-507CL13 MINGW32 /c/temp
$ git clone --mirror git@bitbucket.org:xxxxxx/devices.git
Cloning into bare repository 'devices.git'...
Warning: Permanently added the RSA host key for IP address '2406:da00:ff00::3403:4be4' to the list of known hosts.
git@bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

acordner@PC-AMF-507CL13 MINGW32 /c/temp
$ git clone --mirror https://acordner@bitbucket.org/xxxxxx/devices.git
Cloning into bare repository 'devices.git'...
remote: Counting objects: 1849, done.
remote: Compressing objects: 100% (668/668), done.
fatal: the remote end hung up unexpectedly93 MiB | 4.20 MiB/s
fatal: early EOF
fatal: index-pack failed

acordner@PC-AMF-507CL13 MINGW32 /c/temp
$ git clone --mirror https://acordner@bitbucket.org/xxxxxx/devices.git
Cloning into bare repository 'devices.git'...
remote: Counting objects: 1849, done.
remote: Compressing objects: 100% (668/668), done.
fatal: the remote end hung up unexpectedly16 MiB | 4.56 MiB/s
fatal: early EOF
fatal: index-pack failed

acordner@PC-AMF-507CL13 MINGW32 /c/temp
$ git config --global core.compression 0

acordner@PC-AMF-507CL13 MINGW32 /c/temp
$ git clone --mirror https://acordner@bitbucket.org/xxxxxx/devices.git
Cloning into bare repository 'devices.git'...
remote: Counting objects: 1849, done.
remote: Compressing objects: 100% (668/668), done.
remote: Total 1849 (delta 110), reused 0 (delta 0)
Receiving objects: 100% (1849/1849), 1.88 GiB | 7.08 MiB/s, done.
Resolving deltas: 100% (666/666), done.

and...

acordner@PC-AMF-507CL13 MINGW32 /c/temp/devices.git (BARE:develop)
$ git push --force
Uploading LFS objects: 100% (409/409), 2.0 GB | 563 KB/s, done.
Enumerating objects: 1190, done.
Counting objects: 100% (1190/1190), done.
Delta compression using up to 12 threads
Compressing objects: 100% (752/752), done.
error: RPC failed; HTTP 401 curl 22 The requested URL returned error: 401
fatal: the remote end hung up unexpectedly | 768.00 KiB/s
Writing objects: 100% (986/986), 13.98 MiB | 10.30 MiB/s, done.
Total 986 (delta 415), reused 763 (delta 234)
fatal: the remote end hung up unexpectedly
Everything up-to-date

acordner@PC-AMF-507CL13 MINGW32 /c/temp/devices.git (BARE:develop)
$ git push --force
Uploading LFS objects: 100% (409/409), 2.0 GB | 0 B/s, done.
Enumerating objects: 1190, done.
Counting objects: 100% (1190/1190), done.
Delta compression using up to 12 threads
Compressing objects: 100% (752/752), done.
Writing objects: 100% (986/986), 13.98 MiB | 455.00 KiB/s, done.
Total 986 (delta 415), reused 763 (delta 234)
remote: Resolving deltas: 100% (415/415), completed with 27 local objects.
remote: WARNING Repository size in Bitbucket is currently 1.9 GB
remote: Above the 2 GB size limit further commits are rejected
remote: Reduce repository size by removing large files from history
remote: HINT Learn how to reduce repository size: https://confluence.atlassian.com/x/xgMvEw
To https://bitbucket.org/xxxxxx/devices.git
+ 0cc79af...33f46ec develop -> develop (forced update) 

Now, when I go to clone the repo, I get this message:

and when I click Yes, I get:

So I open a terminal window and enter git lfs install manually and the hooks are added.

That being said, I think this is resolved.