git push error- on git server

[Mangesh Chougule]

$ git push

Enumerating objects: 5, done.

Counting objects: 100% (5/5), done.

Delta compression using up to 8 threads

Compressing objects: 100% (3/3), done.

fatal: Out of memory, malloc failed (tried to allocate 4168062027 bytes)

fatal: sha1 file '<stdout>' write error: Broken pipe

error: remote unpack failed: unpack-objects abnormal exit

I am trying to push *.sql which is around 4Gb.

Can you help me get over this issue? Is there any limitation on the file size in the repository?

[Rafael Pinto Sperafico]

Hi @Mangesh Chougule ,

Thank you for taking the time and provide us with a bit more of context on the problem you are facing.

The suggestion will be the same as previous mentioned, however, I will be providing you with more explanation as to why you should follow the suggested.

You are attempting on pushing a file (3.8Gb in size) to a repository. If you are not using Git LFS, whenever user or CI tool that will pull from this repository will subsequently request that 3.8Gb SQL file. This will cause a great load on your Bitbucket DataCenter when "git-upload-pack" gets issued on the server and all of this could be avoid if you push larger files using Git LFS.

As pointed out ( https://www.atlassian.com/git/tutorials/git-lfs ) Git clients will be able to fetch/pull from your repository and only when needed they would be issuing a Git LFS command to retrieve those larger files.

On top of that, if not only your Git client but also the Git version installed and used by Bitbucket DataCenter is 2.18.0 or above, you can benefit from SSH Protocol v2 (this feature became available on Bitbucket v5.13 or above - Bitbucket Server 5.13 release notes).

It is best practises and recommended using SSH over HTTP(S) protocol when working with larger repositories, because the SSH protocol is much more reliable and it is meant for this purpose. Please, review Using SSH keys to secure Git operations documentation as it provides you with necessary information as:

• enable SSH access in Bitbucket Server
• create SSH keys

Just as a reminder, the size of your repository should be given by the instructions provided on Git Repository Size from the Command Line. The command "du -hs" does not actually give the size of a repository.

If SSH is not an option, then you should be reviewing:

• memory allocation during the "git push" operation not only in your Windows client but also in Bitbucket DataCenter.
• if you have Windows Defender or any Firewall between your machine and Bitbucket DataCenter that could be contributing to your push to not succeed.
• .gitconfig settings ( https://git-scm.com/docs/git-config ) so you can tweak Git's configuration to handle the larger file you are attempting on pushing.

Kind regards,
Rafael

[Mangesh Chougule]

Hi Rafel,

Thanks for the response.

Im trying to push 3.8 GB file to a git server from my windows mahine using git bash.

1. git version 2.23.0.windows.1

2. Windows - 64 bit

3.Ram = 16gb

4. $ du -hs >> 4.7G

5. Data centre- GIT server is hosted.

6. What is the protocol you are using (HTTP(S) or SSH)?

Ans: HTTPS

[Rafael Pinto Sperafico]

Hi @Mangesh Chougule ,

Based on the error provided, we are unable to confirm if you "remote" to where you are running the git push command is defined as HTTP(S) or SSH protocol. For larger files, you should be considering using:

• SSH protocol
• Git LFS (why pushing larger files to LFS - https://www.atlassian.com/git/tutorials/git-lfs)

If you really need to have this file available to your repository and not to a Git LFS (Large File Storage) then you could play with ".gitconfig" settings related to pack (https://git-scm.com/docs/git-config):

...
[pack]
  threads = 1
  deltaCacheSize = 128m
  packSizeLimit = 128m
  windowMemory = 128m

[core]
  packedGitLimit = 128m
  packedGitWindowSize = 128m

So we can have a bit more of context:

• What Git version you are running? (git --version)
  
• What platform is your machine (Unix/Windows/32/64 Bit)?
• How much memory RAM do you have available on your machine?
• How big is the repository? (check command on Git Repository Size from the Command Line)
• Where is your repository server hosted on (Server - Bitbucket Server or Bitbucket DataCenter / Cloud - Bitbucket Cloud, GitHub)?
• What is the protocol you are using (HTTP(S) or SSH)?

Perhaps you should consider reading about Scaling Bitbucket Server in which provides examples about Memory and CPU consumption based the way you work with your repository. Furthermore, a comparison between HTTP(S) vs SSH protocol.

Kind regards,
Rafael