Deleting files to reduce repo size

[Nico Kanitz]

Dear Community,

lately the warning saying that my repository size is over 1GB has been jumping anytime I access bitbucket.

Then I followed the instructions given by the link:

https://confluence.atlassian.com/bitbucket/reduce-repository-size-321848262.html

I tried by using the git filter-branch to rewrite history, but everytime I try deleting a file by name then I followed the Garbage collecting dead data instructions but after all if I list again the big files in my repo by running the git_find_big.sh script the file is still there and I am still getting the warning.

The thing is that I am still getting the warning but when I run

git count-objects -Hv

It says that the repo is only 600 MiB (before was 1.3GB)

The Whole history rewriting is taking like an hour or so, and I when I push I am pushing a thousands of commits therefore I guess I am doing the right thing but I am not sure, could anyone give me hint?

Thanks in advance

[Ana Retamal · Accepted solution]

Hi Nico!

Are you sure you've removed that big file from all the commits where it appears?

Also, could you let us know the name of the repo so we can have a look? 

Kind regards,

Ana

[Nico Kanitz]

The repo is uphillgmbh/fanpagekarma

After removing files again, we now see the size of the project went down to 576MB, can you confirm that size?

Thanks

[Ana Retamal]

I've checked the size on the remote and I can also see 708 MB, is not exactly the same as what you showed, but it could be caused by some changes you added recently, new references, etc . Is there anything else we can help you with?

Cheers!

Ana

[Nico Kanitz]

Thanks for your reply. We still working on this, but it seems we already reduced the size a bit...

[Mahender.Bogi]

We are trying same case here for our repo, but its not helping at all. We have cleaned up 200MB file & have run rewrite history. But its not reducing the size. Are we doing something wrong here?

[Ana Retamal]

Hi @Mahender.Bogi , I've checked your account and I couldn't find any big repo. Can you let us know the name of the repo you're referring to? 

Also, you said you've run rewrite history, can you tell us which command you ran exactly?

Cheers,

Ana

[Praveen Kumar Voruganti]

Hi Ana,

We followed below commands

1. forked new repo <repository-url>
2. removed the file 193MB zip file and committed but not pushed as mentioned
3. ran the bfg delete file
$ java -jar bfg.jar --delete-files <200mbfilename>
4. ran below command
$ git -c gc.auto=1 -c gc.autodetach=false -c gc.autopacklimit=1 -c gc.garbageexpire=now -c gc.reflogexpireunreachable=now gc --prune=all
5. pushed the changes by below command
$ git push --all --force && git push --tags --force
after push the changes.
6. I raised a ticket to run the serverside GC cleanup. after serverside GC cleanup completes size is not reducing.

please let me know what step i am missing.

[Mahender.Bogi]

Hi Ana,

Please check the repository under hostanalytics account. And praveen above mentioned the steps we did. 

Appreciate your help here.

Thanks,

-Mahender

[Nico Kanitz]

Hi Ana,

our project again show a size bigger than 2GB. 

Running git count-objects -Hv shows the following results:

count: 2334
size: 169.12 MiB
in-pack: 188303
packs: 48
size-pack: 633.48 MiB
prune-packable: 38
garbage: 0
size-garbage: 0 bytes

Could you please run the GC on our repo: uphillgmbh/fanpagekarma

Thanks in advance 
Nicolas

[Ana Retamal]

Hi Nicolas! 

I've just checked and the repo size is ~740MB, which looks about right. Seems like the garbage collection was run automatically before I could see your message. 

Let us know if you need anything else,

Ana

[Nico Kanitz]

Dear Ana,

thanks! Yes, it looks like that happend. 

Could you give me some insights why a repository can grow from 1GB to 2GB within a day, if the size itself is about 740 MB?

Thanks again
Nicolas

[Nico Kanitz]

Hi Ana,

our project again show a size close to 2GB. 

Running git count-objects -Hv shows the following results:

count: 1498
size: 11.55 MiB
in-pack: 205417
packs: 47
size-pack: 812.75 MiB
prune-packable: 92
garbage: 0
size-garbage: 0 bytes

Could you please run the GC on our repo: uphillgmbh/fanpagekarma

Thanks in advance 
Nicolas

[Tyler T]

Hi Nicolas, the size is back down again. It is possible that whatever large object is causing the huge spike in memory is still in the git history of someone pushing to the repository.

[Nico Kanitz]

Hi Tyler, 

thanks for your feedback. We suspect that the GC just does not run often enough, and it's better immediately afterwards. Without any change from our side.

Best

Nicolas

[Nico Kanitz]

Hi Tyler,

our project again show a size close to 2GB. 

Running git count-objects -Hv shows the following results:

count: 18586
size: 138.18 MiB
in-pack: 255546
packs: 546
size-pack: 934.82 MiB
prune-packable: 999
garbage: 2
size-garbage: 198.97 MiB

Could you please run the GC on our repo: uphillgmbh/fanpagekarma ?

Thanks in advance 
Nicolas

[Nico Kanitz]

Hi Team,

our project again show a size close to 2GB. 

Running git count-objects -Hv shows the following results:

count: 3594
size: 21.14 MiB
in-pack: 303254
packs: 168
size-pack: 875.38 MiB
prune-packable: 286
garbage: 2
size-garbage: 198.97 MiB

Could you please run the GC on our repo: uphillgmbh/fanpagekarma ?

Thanks in advance 
Nicolas

[Tyler T]

@Nico Kanitz done!

[Nico Kanitz]

Hi Team,

our project again show a size close to 2GB. 

Running git count-objects -Hv shows the following results:

count: 6672
size: 47.82 MiB
in-pack: 320404
packs: 304
size-pack: 913.43 MiB
prune-packable: 625
garbage: 2
size-garbage: 198.97 MiB

Could you please run the GC on our repo: uphillgmbh/fanpagekarma ?

Thanks in advance 
Nicolas

[Nico Kanitz]

@Tyler T Could you please check the comment before?

Thanks in advance

[Nico Kanitz]

or @Ana Retamal  ?

[Ana Retamal]

Hi @Nico Kanitz,

I've run git gc on your repo and the size has decreased to 836.1 MB.

Have a  good day! :) 

Ana 

[Nico Kanitz]

@Ana Retamal 

Hi Team,

our project again show a size close to 2GB. 

Running git count-objects -Hv shows the following results:

count: 10010
size: 74.53 MiB
in-pack: 333074
packs: 471
size-pack: 955.34 MiB
prune-packable: 928
garbage: 2
size-garbage: 198.97 MiB

Could you please run the GC on our repo: uphillgmbh/fanpagekarma ?

Thanks in advance 
Nicolas

[Nico Kanitz]

or @Tyler T ?

[Tyler T]

Hi @Nico Kanitz 

Looks like GC ran automatically 15 hours ago.. I see the size as 941.6 MB