Unable to push: out of memory loading unknown object

[Tom Kronmiller]

Attempting to push a branch to my central server fails, with the error message "Out of memory loading unknown object".  There is about 60GB RAM free on the git server, plus 144GB free swap space so I am skeptical that it is really out of virtual memory.  The atlbitbucket java processes have their resource limit on address space set to "unlimited".

Any suggestions on what to check or how to fix this?  Are there any git or bitbucket configuration parameters which might cause this out of memory by being too low?

$ git pull
Already up to date.

$ git status
On branch xxxxx
Your branch is ahead of 'origin/xxxxx' by 871 commits.
(use "git push" to publish your local commits)

$ git push
Enumerating objects: 2, done.
Counting objects: 100% (2/2), done.
Delta compression using up to 12 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 504 bytes | 504.00 KiB/s, done.
Total 2 (delta 0), reused 0 (delta 0)
remote: error: Out of memory loading unknown object
remote:
remote: Fetching revisions from SVN repository:
remote: up to date
remote: Sending commits to SVN repository:
To http://githost:7990/yyyyy.git
! [remote rejected] xxxxx -> xxxxx (pre-receive hook declined)
error: failed to push some refs to 'http://release@githost:7990/yyyyy.git'

The atlassian-bitbucket.log shows two messages at the point of the failure, one being:

2019-01-22 15:23:37,434 ERROR [sg:interactive-398] o.e.j.i.storage.file.ObjectDirectory ERROR: Exception caught while accessing pack file /mnt/ssd-raid/var-atlassian/application-data/bitbucket/shared/data/repositories/13/objects/pack/pack-a0339390e4c009d442fe05eeec75d1e8592a2cbb.pack, the pack file might be corrupt, {1}. Caught {2} consecutive errors while trying to read this pack.
java.io.IOException: In-memory buffer limit exceeded
at org.eclipse.jgit.internal.storage.file.PackFile.load(SourceFile:880)
at org.eclipse.jgit.internal.storage.file.PackFile.get(SourceFile:293)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(SourceFile:482)

[... many more stack items skipped...]

The other one is:

2019-01-22 15:23:37,783 INFO [hook-callback:thread-2] release @PVJGGGx923x4275x0 10.16.0.85 "POST /yyyyy.git/git-receive-pack HTTP/1.1" c.a.s.i.h.r.DefaultRepositoryHookService [zzzzz[13]] hook 'svnmirror-hook' vetoed the push request

[Tom Kronmiller · Accepted solution]

The following procedure seems to be what worked around the issue.

1. Log into the central server as the user who owns the git installation.
2. Locate the repository, here .../application-data/bitbucket/shared/data/repositories/13
3. Run "git repack -a --max-pack-size=256m", which could take multiple hours (it did for me).

After that I went back to the sandbox which couldn't push and tried again. The sandbox now needed a fresh pull plus the workaround for the "'xxxx@<revision>' is out of date: it has been modified externally" issue (https://community.atlassian.com/t5/Bitbucket-questions/How-do-I-fix-quot-trunk-45967-is-out-of-date-it-has-been/qaq-p/975694), but after that a push was able to complete. (It took about 15 minutes to run, which may or may not be related.)