Git LFS push fails for all our repositories

[Kevin Baynham]

All of a sudden, any files that are tracked via Git LFS can no longer be pushed. All operations were working fine yesterday, but from this morning onward this issue is occurring with all of our repositories and on all of our computers - it is not localized to a particular computer/repo.

Note that pulling standard/LFS files works correctly and pushing files that are not tracked via LFS is also working correctly. When removing the problematic files from LFS tracking these files can be pushed successfully without LFS.

We were using Sourcetree 3.0.1 (Mac) and have updated to 3.2.1, but to no avail.

We can only assume that there has been some server/policy changes (yet we cannot find any announcements) or there are currently server issues (but the status page shows that all systems are operational). 

We have gotten around the problem by disabling LFS, but this is a major issue as we have many projects that are using LFS and can no longer be maintained without changes. Any advice would be greatly appreciated.

Here are logs that may help clarify the situation:

Sourcetree console output

git --no-optional-locks -c color.branch=false -c color.diff=false -c color.status=false -c diff.mnemonicprefix=false -c core.quotepath=false -c credential.helper=sourcetree -c lfs.customtransfer.bitbucket-media-api.path=/Applications/Sourcetree.app/Contents/Resources/git_local/git-lfs/git-lfs-bitbucket-media-api push -v --tags --set-upstream origin refs/heads/test:refs/heads/test
Pushing to https://KevinBaynham@bitbucket.org/fuzzy-logic-studio/unity-sasolmymining.git
Fatal error: Server error: https://bitbucket.org/%7B%7D/%7B3f69e91e-3910-497f-9665-7ee84fc3b307%7D/info/lfs/object/verify?upload_id=4a8ab33f-12a0-4a8c-b25c-13e51aeb1b75
Uploading LFS objects: 0% (0/1), 3.5 KB | 0 B/s, done

Errors logged to /Users/kevin/Documents/Projects/unity-sasolmymining/.git/lfs/logs/20190813T161623.941746.log
Use `git lfs logs last` to view the log.
error: failed to push some refs to 'https://KevinBaynham@bitbucket.org/fuzzy-logic-studio/unity-sasolmymining.git'
Completed with errors, see above

Git LFS Log
./.git/lfs/logs/20190813T161623.941746.log

git-lfs/2.6.1 (GitHub; darwin amd64; go 1.11.2; git dc072c3e)
git version 2.20.1

$ git-lfs pre-push origin https://KevinBaynham@bitbucket.org/fuzzy-logic-studio/unity-sasolmymining.git
Fatal error: Server error: https://bitbucket.org/%7B%7D/%7B3f69e91e-3910-497f-9665-7ee84fc3b307%7D/info/lfs/object/verify?upload_id=0d607db0-16ba-460b-bbb3-d49f26122b3d

Server error: https://bitbucket.org/%7B%7D/%7B3f69e91e-3910-497f-9665-7ee84fc3b307%7D/info/lfs/object/verify?upload_id=0d607db0-16ba-460b-bbb3-d49f26122b3d
github.com/git-lfs/git-lfs/errors.Errorf
/Users/bk2204/checkouts/git-lfs/errors/errors.go:69
github.com/git-lfs/git-lfs/lfshttp.defaultError
/Users/bk2204/checkouts/git-lfs/lfshttp/errors.go:119
github.com/git-lfs/git-lfs/lfshttp.(*Client).handleResponse
/Users/bk2204/checkouts/git-lfs/lfshttp/errors.go:51
github.com/git-lfs/git-lfs/lfshttp.(*Client).DoWithRedirect
/Users/bk2204/checkouts/git-lfs/lfshttp/client.go:305
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithCreds
/Users/bk2204/checkouts/git-lfs/lfsapi/auth.go:96
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithAuth
/Users/bk2204/checkouts/git-lfs/lfsapi/auth.go:67
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoWithAuth
/Users/bk2204/checkouts/git-lfs/lfsapi/auth.go:25
github.com/git-lfs/git-lfs/tq.verifyUpload
/Users/bk2204/checkouts/git-lfs/tq/verify.go:54
github.com/git-lfs/git-lfs/tq.(*basicUploadAdapter).DoTransfer
/Users/bk2204/checkouts/git-lfs/tq/basic_upload.go:147
github.com/git-lfs/git-lfs/tq.(*adapterBase).worker
/Users/bk2204/checkouts/git-lfs/tq/adapterbase.go:176
runtime.goexit
/usr/local/Cellar/go/1.11.2/libexec/src/runtime/asm_amd64.s:1333
Fatal error
github.com/git-lfs/git-lfs/errors.newWrappedError
/Users/bk2204/checkouts/git-lfs/errors/types.go:184
github.com/git-lfs/git-lfs/errors.NewFatalError
/Users/bk2204/checkouts/git-lfs/errors/types.go:228
github.com/git-lfs/git-lfs/lfshttp.(*Client).handleResponse
/Users/bk2204/checkouts/git-lfs/lfshttp/errors.go:66
github.com/git-lfs/git-lfs/lfshttp.(*Client).DoWithRedirect
/Users/bk2204/checkouts/git-lfs/lfshttp/client.go:305
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithCreds
/Users/bk2204/checkouts/git-lfs/lfsapi/auth.go:96
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithAuth
/Users/bk2204/checkouts/git-lfs/lfsapi/auth.go:67
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoWithAuth
/Users/bk2204/checkouts/git-lfs/lfsapi/auth.go:25
github.com/git-lfs/git-lfs/tq.verifyUpload
/Users/bk2204/checkouts/git-lfs/tq/verify.go:54
github.com/git-lfs/git-lfs/tq.(*basicUploadAdapter).DoTransfer
/Users/bk2204/checkouts/git-lfs/tq/basic_upload.go:147
github.com/git-lfs/git-lfs/tq.(*adapterBase).worker
/Users/bk2204/checkouts/git-lfs/tq/adapterbase.go:176
runtime.goexit
/usr/local/Cellar/go/1.11.2/libexec/src/runtime/asm_amd64.s:1333

Current time in UTC: 
2019-08-13 12:49:40

ENV:
LocalWorkingDir=/Users/kevin/Documents/Projects/unity-sasolmymining
LocalGitDir=/Users/kevin/Documents/Projects/unity-sasolmymining/.git
LocalGitStorageDir=/Users/kevin/Documents/Projects/unity-sasolmymining/.git
LocalMediaDir=/Users/kevin/Documents/Projects/unity-sasolmymining/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/kevin/Documents/Projects/unity-sasolmymining/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/kevin/Documents/Projects/unity-sasolmymining/.git/lfs
AccessDownload=basic
AccessUpload=basic
DownloadTransfers=basic,bitbucket-media-api
UploadTransfers=basic,bitbucket-media-api
GIT_CONFIG_PARAMETERS='color.branch=false' 'color.diff=false' 'color.status=false' 'diff.mnemonicprefix=false' 'core.quotepath=false' 'credential.helper=sourcetree' 'lfs.customtransfer.bitbucket-media-api.path=/Applications/Sourcetree.app/Contents/Resources/git_local/git-lfs/git-lfs-bitbucket-media-api'
GIT_EXEC_PATH=/Applications/Sourcetree.app/Contents/Resources/git_local/libexec/git-core
GIT_OPTIONAL_LOCKS=0
GIT_PAGER=cat
GIT_PREFIX=

[Kevin Baynham]

This issue has resolved itself.

In the end, we were unable to push LFS-tracked files for at least 10 hours (beginning of the working day until closing), and when returning to work the following day the service was working as per usual.

I want to reinforce that I still feel that this was not an issue on our side, and the following day I had found out that our branch in a different city (also within South Africa) had the same problem, and they obviously run a different network with a different ISP.

This was very frustrating and problematic as we have now missed a deadline due to this interruption. If this was not an Atlassian server problem I apologize, but I feel that it was very poor not to announce the issue to your customers nor publish a notification after the fact to say that it had been resolved.