Delete large file from the repository

[sunny386]

Hi, 

I have a repository named "attachmentparenting.d8.org" and there is a folder called "audio" that I need to remove. It contains lots of big mp3 files and they were committed mistakenly.

I need to decrease the repository size, once I tried to delete the large file from another repository but end up losing some data and I don't want to take any risk this time.

anyone can help me to achieve this without any issue?

Thanks in advance,

Sunny

[Syahrul · Accepted solution]

Hi @sunny386

If I got this correct you wish to delete the "Audio" folder in your repositories and do not wish to break or lose any other data on your repositories?

If yes, can you follow these steps to delete the "Audio" folder from your repositories:

1. Download BFG repo cleaner 
2. Clone your bare repositories by using --mirror options
   

   git clone --mirror https://bitbucket.org/<workspace>/<reponame>

   It's important to use --mirror options to clone so you are deleting the files on bare repositories. You can also copy and paste the whole folder somewhere else as a backup.
3. Copy the BFG tools and place them outside the clone bare repositories
4. Open terminal or CMD and run the following commands to delete the folder. 
   

   java -jar bfg-1.14.0.jar --delete-folders <foldername> <reponame>

   Folder name = Folder name you want to delete
   Repo name = repositories folder name that you clone on Step 2
5. Once complete, we need to clean up the repositories with:
   

   cd <reponame>
   git reflog expire --expire=now --all && git gc --prune=now --aggressive

6. When the cleanup is done you can use git push to push the changes over to remote and reach out to support to request for Git GC to refresh the remote repositories.
   
   

Additional note, it's best to create a new folder and clone the bare repositories in it and put the BFG in the new folder. This is to avoid mistakenly deleting duplicate non-bare repositories.

I hope this helps, please let me know if you have any questions.

Cheers,
Syahrul

[sunny386]

Hi @Syahrul Thank you for the response, 

I have followed all the steps given but somehow while pushing it gives this error.

"error: RPC failed; curl 55 OpenSSL SSL_write: Connection was aborted, errno 10053
fatal: the remote end hung up unexpectedly
Writing objects: 100% (19810/19810), 1.68 GiB | 575.00 KiB/s, done.
Total 19810 (delta 4262), reused 4262 (delta 4262), pack-reused 15548
fatal: the remote end hung up unexpectedly

Everything up-to-date"

Is it because of my internet connection and the repo size is big? if you run the git GC now will it help me to reduce the space and remove the audio folder? the audio folder is still showing in the repo and the size is also the same.

Please suggest anything that works for me.

Thank you.

[Syahrul]

Hi @sunny386

Before pushing can you run "git count-objects -vH" inside the repo, if the size is less than 1.8GB then the files are properly deleted and you may need to use "git push -f" to force the push.

I did run a GC on the affected repositories, but the size still remains at 1.8GB so I believe the latest push did not go through, could you try again with the force option?

If it's still failing could you restart the steps again and share with me the output of each command? I wish to verify if there are any failed steps that may affect the removal of the folders.

I look forward to hearing from you.

Cheers,
Syahrul

[sunny386]

Hi @Syahrul , This command (git count-objects -vH) is showing the same size (size-pack: 1.74 GiB)

Please check the below screenshot of every step that I followed again on the folder that I copied before running any command.

It seems like the git push isn't working again. I tried with -f too but no luck.

Thank you.

[sunny386]

@Syahrul , I have manually deleted every file from the audio folder, will this help to reduce the repo size? as audio folder isn't visible in the repo now. 

[sunny386]

Hi @Syahrul , Please check the given screenshots below, I followed all steps again after deleting the audio folder manually. It looks like git push also works fine this time, Can you run git GC one more time, please?

[sunny386]

Hi @Syahrul , I am still having the same issue and waiting for your response. Please help me out. Thank you.

[Syahrul]

Hi @sunny386

Thanks for the update. It looks like your second force push did the work since running an aggressive GC on the affected repositories reduced the repositories size accordingly.

Can you check and confirm this with me and let me know if the size is still showing large or if the files still exist in the remote repositories.

I look forward to hearing from you

Cheers,
Syahrul

[sunny386]

Hi @Syahrul

Yes the size is decreased.

Thank you for the help.