Unable to push to an empty repository

[peakman]

There are many people reporting similar problems and I think I've tried all the proposed solutions without success.

Whenever I try I receive the message: 

client_loop: send disconnect: Broken pipe

send-pack: unexpected disconnect while reading sideband packet

I know the empty repository exists.

I know my credentials are correct because I can push successfully to other repositories in the same project.

I've tried both SSH and HTTP but the result is the same.

I'm using Sourcetree v 4.1.3 but I get the same result using git v 2.33.0 from the command line. I use macOS BigSur 11.6.1.

I've been using BitBucket and Sourcetree for several years and have been able to push to empty repositories in the past.  Any ideas what might have changed and why I can do so no longer?

One thing that changed at some time fairly recently is the ability to choose the name of the main branch.  I have tried using "main" and "master" but with no success.

[Theodora Boudale · Accepted solution]

Hi @peakman,

1. What is the output of the following command in your local repo?

git count-objects -Hv

That will show us the size of the repository.

2. Is the repo using Git LFS or not?

3. If you're still using SSH now, could you please provide us with the output of the following command, which will give more verbose output?

GIT_TRACE_PACKET=1 GIT_TRACE=1 GIT_SSH_COMMAND="ssh -vvv" git push <repo_url>

Please make sure to sanitize any private/sensitive info in the output like workspace or repo names, prior to sharing it here.

Kind regards,
Theodora

[peakman]

Thanks for response. Specific answers below.  

More generally, I am now using SourceTree 4.1.5.

I have created a new repository similar to the one below that fails.  I was able to push the new repository to an empty repository.

1. OUTPUT FROM git count-objects -Hv
==================================
count: 581
size: 20.00 MiB
in-pack: 0
packs: 0
size-pack: 0 bytes
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

2. USING Git LFS
==============

No

3. TRACE
=========

~/Development/myrepo  GIT_TRACE_PACKET=1 GIT_TRACE=1 GIT_SSH_COMMAND="ssh -vvv" git push git@bitbucket.org:peakman/emptyrepo.git
16:57:24.114182 git.c:455 trace: built-in: git push git@bitbucket.org:peakman/emptyrepo.git
16:57:24.116845 run-command.c:666 trace: run_command: unset GIT_PREFIX; 'ssh -vvv' git@bitbucket.org 'git-receive-pack '\''peakman/emptyrepo.git'\'''
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/peakman/.ssh/config
debug1: /Users/peakman/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to bitbucket.org port 22.
debug1: Connection established.
debug1: identity file /Users/peakman/.ssh/id_rsa type 0
debug1: identity file /Users/peakman/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version conker_a7531beec7 937fb3471f96
debug1: no match: conker_a7531beec7 937fb3471f96
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bitbucket.org:22 as 'git'
debug3: hostkeys_foreach: reading file "/Users/peakman/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/peakman/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-s
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug3: hostkeys_foreach: reading file "/Users/peakman/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/peakman/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: hostkeys_foreach: reading file "/Users/peakman/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/peakman/.ssh/known_hosts:60
debug3: load_hostkeys: loaded 1 keys from 104.192.141.1
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /Users/peakman/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/peakman/.ssh/id_rsa RSA SHA256:....... explicit agent
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-dss,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ssh-rsa-cert-v01@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/peakman/.ssh/id_rsa RSA SHA256:dPclQsDFDjsqYFN0kieybzxuu6pV+c/lTZMjiWfJmB4 explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/peakman/.ssh/id_rsa RSA SHA256:dPclQsDFDjsqYFN0kieybzxuu6pV+c/lTZMjiWfJmB4 explicit agent
debug3: sign_and_send_pubkey: RSA SHA256:dPclQsDFDjsqYFN0kieybzxuu6pV+c/lTZMjiWfJmB4
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to bitbucket.org ([104.192.141.1]:22).
debug2: fd 6 setting O_NONBLOCK
debug2: fd 7 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x20
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env .....
debug1: Sending env LANG = en_GB.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env ...
debug1: Sending command: git-receive-pack 'peakman/emptyrepo.git'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
16:57:25.381063 pkt-line.c:80 packet: push< 0000000000000000000000000000000000000000 capabilities^{}\0report-status report-status-v2 delete-refs side-band-64k quiet atomic ofs-delta object-format=sha1 agent=git/2.32.0
16:57:25.381541 pkt-line.c:80 packet: push< 0000
16:57:25.381607 pkt-line.c:80 packet: push> 0000000000000000000000000000000000000000 5a472190e62e99b97d9a86218aeefe920e6a0f3b refs/heads/main\0 report-status-v2 side-band-64k object-format=sha1 agent=git/2.33.0
16:57:25.381617 pkt-line.c:80 packet: push> 0000
16:57:25.381741 run-command.c:666 trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
16:57:25.386581 git.c:455 trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 536, done.
Counting objects: 100% (536/536), done.
Delta compression using up to 8 threads
debug2: channel 0: rcvd adjust 173
Compressing objects: 100% (518/518), done.
debug3: send packet: type 1536)
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i0/1744896 o0/0 e[write]/0 fd 6/7/8 sock -1 cc -1)

send-pack: unexpected disconnect while reading sideband packet
debug1: fd 0 clearing O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
Connection to bitbucket.org closed by remote host.
Transferred: sent 364172, received 2328 bytes, in 0.6 seconds
Bytes per second: sent 605711.6, received 3872.1
debug1: Exit status -1
fatal: the remote end hung up unexpectedly

[peakman]

After further investigation (by incrementally  re-building the repo that failed to push) I think I have got an explanation about what causes the failure.  

I started with a basic repo that pushed successfully and then added files to it.  All went well for a few iterations until I added some .png files.  At this point I got the error messages illustrated by the post above.

I used to be able to push binary files but at some point that seems to have changed. My failing repo includes binary files that were clearly pushed in the past. I think something must have changed at Bitbucket.

[Theodora Boudale]

Hi @peakman,

Thank you for your reply and info.

Could you please let me know
- what is the approximate number of .png files you are trying to push when you start getting this error
- the size of these files

We recommend using Git LFS if you need to version binary files, in order to have better performance.

I have nonetheless attempted to reproduce what you are reporting, but I haven't been able to so far, and it would be useful to know the info I asked for above.

Kind regards,
Theodora

[peakman]

I tried to add only 3 .png files.

Their combined size was less than 500Kb.

I've now created an even simpler test case.

1. I created a new repo with a README and a gitignore.

2.I cloned it.

3.I updated the README locally and committed and pushed.  This worked.

4.I added a single small pdf and committed and pushed.  The push failed.

I've never used LFS as I don't think I should need it.  I have therefore never installed the Git LFS Client.  

[Theodora Boudale]

Hi @peakman

Thank you for the info. I have attempted to reproduce this issue by pushing .png and .pdf files of that approximate size and then with larger ones as well, using different computers and networks, but I haven't been able to.

I suspect that issue may be related to your network, or to your computer, or to its environment.

My suggestion would be to try the following in order to isolate the source of the problem:
- try the same push using the same computer, but a different network, e.g. a 4G network
- if you have access to a different computer, try the same push from that computer on the same network

- You can try adding the following in the ~/.ssh/config file of your machine to check if this will help keep the SSH connection alive for longer:

Host bitbucket.org
    ServerAliveInterval 300
    ServerAliveCountMax 2

This will make your SSH client send a null packet to Bitbucket every 300 seconds (5 minutes) and give up and give up if it doesn't receive a response after 2 times. You can set these to a higher value if the push needs longer to complete.

- There is a possibility that you receive this error due to Anycast Shift, you can read more details in the following article:

https://community.atlassian.com/t5/Bitbucket-articles/Diagnosing-and-Troubleshooting-TCP-Resets-from-Bitbucket-Cloud/ba-p/1562132

You can try to add Bitbcuket's legacy IP address to your hosts file as explained in the article to see if that resolves the issue. If so, please note that this should be a temporary workaround as mentioned in the article.

- We've had customers report also that a firewall or some other software, e.g. antivirus or some VPN client configuration was breaking the connection.

- We also have the following guide on troubleshooting network issues, which may be useful:

• https://confluence.atlassian.com/bbkb/troubleshooting-network-issues-389778693.html

Kind regards,
Theodora

[peakman]

Thanks for the additional info.  In the meantime I continued to try things that would enable me to continue pushing my main working repo without getting an error.

Eventually I succeeded by cloning the latest repo I was able to push to Bitbucket. I then rebuilt on top of the clone. I'd tried similar approaches before without success but suddenly I was able to push again.  What a relief!

It's all rather unsatisfactory because I never pinned down the root cause of the problem.  I think my network is probably good and I rather suspect that I've somehow managed to corrupt my local git such that it works fine on my machine but will not push.  I need to make progress now so I'll park your latest suggestions unless or until I hit another problem.

Thanks again for your help.

[peakman]

I did hit the problem again.  My repo, which was pushing successfully, suddenly stopped doing so after just a minor change.

I changed to 4G network as you suggested and it worked again immediately. My normal network, which is usually fine, obviously has problems that persist for  some time.  The only time they cause me a problem is with git push.

Thanks yet again for all your help.

[Theodora Boudale]

Hi @peakman,

Thank you for the update and you are very welcome.

It's good to hear that there were no issues when using the 4G network.

If you wish to troubleshoot this further, you can refer to our guide "Troubleshooting Network Issues" which I posted above for some ideas on possible causes. Your ISP may also be able to help resolve this issue, as they have access to your network.

Please feel free to reach out if you ever need anything else.

Kind regards,
Theodora