Suddenly pull/push does not work

[İsa aşcı]

My ssh key is correctly in place. It's been there for some while. Today suddenly fetch and pull does not work. I tried to debug but noting more interesting than the standart error. Anyone experienced the same lately?

Here's my log:

GIT_SSH_COMMAND="ssh -vvv" git pull
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/isaasci/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/isaasci/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/isaasci/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to bitbucket.org port 22.
debug1: Connection established.
debug1: identity file /Users/isaasci/.ssh/id_rsa type 0
debug1: identity file /Users/isaasci/.ssh/id_rsa-cert type -1
debug1: identity file /Users/isaasci/.ssh/id_ecdsa type 2
debug1: identity file /Users/isaasci/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/isaasci/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/isaasci/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/isaasci/.ssh/id_ed25519 type -1
debug1: identity file /Users/isaasci/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/isaasci/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/isaasci/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/isaasci/.ssh/id_xmss type -1
debug1: identity file /Users/isaasci/.ssh/id_xmss-cert type -1
debug1: identity file /Users/isaasci/.ssh/id_dsa type -1
debug1: identity file /Users/isaasci/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version conker_690f9b85d2 679e046dca26
debug1: compat_banner: no match: conker_690f9b85d2 679e046dca26
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bitbucket.org:22 as 'git'
debug3: record_hostkey: found key type ED25519 in file /Users/isaasci/.ssh/known_hosts:65
debug3: record_hostkey: found key type ECDSA in file /Users/isaasci/.ssh/known_hosts:66
debug3: record_hostkey: found key type RSA in file /Users/isaasci/.ssh/known_hosts:67
debug3: load_hostkeys_file: loaded 3 keys from bitbucket.org
debug1: load_hostkeys: fopen /Users/isaasci/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-s
debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-256,rsa-sha2-512
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:ybgmFkzwOSotHTHLJgHO0QN8L0xErw6vd0VhFA9m3SM
debug3: record_hostkey: found key type ED25519 in file /Users/isaasci/.ssh/known_hosts:65
debug3: record_hostkey: found key type ECDSA in file /Users/isaasci/.ssh/known_hosts:66
debug3: record_hostkey: found key type RSA in file /Users/isaasci/.ssh/known_hosts:67
debug3: load_hostkeys_file: loaded 3 keys from bitbucket.org
debug1: load_hostkeys: fopen /Users/isaasci/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'bitbucket.org' is known and matches the ED25519 host key.
debug1: Found key in /Users/isaasci/.ssh/known_hosts:65
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 3 keys
debug1: Will attempt key: /Users/isaasci/.ssh/id_rsa RSA SHA256:gxuV7n2lINJxAziDbTtbVWBg2FcJRC5wQ/ykPfgC78M agent
debug1: Will attempt key: /Users/isaasci/.ssh/id_ecdsa ECDSA SHA256:MItks7k4VMKSN7MbH70qDoSpoRgmmK/3Xee2aZ3G56c agent
debug1: Will attempt key: isaasci@192.168.1.35 ED25519 SHA256:MGgjWLrCQpfF/jakArgdNF4FAvuM8yNPgLc9JYFUOvw agent
debug1: Will attempt key: /Users/isaasci/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/isaasci/.ssh/id_ed25519
debug1: Will attempt key: /Users/isaasci/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/isaasci/.ssh/id_xmss
debug1: Will attempt key: /Users/isaasci/.ssh/id_dsa
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ssh-rsa,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp521,rsa-sha2-256,ecdsa-sha2-nistp384,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/isaasci/.ssh/id_rsa RSA SHA256:gxuV7n2lINJxAziDbTtbVWBg2FcJRC5wQ/ykPfgC78M agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/isaasci/.ssh/id_rsa RSA SHA256:gxuV7n2lINJxAziDbTtbVWBg2FcJRC5wQ/ykPfgC78M agent
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:gxuV7n2lINJxAziDbTtbVWBg2FcJRC5wQ/ykPfgC78M
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:gxuV7n2lINJxAziDbTtbVWBg2FcJRC5wQ/ykPfgC78M
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to bitbucket.org ([104.192.141.1]:22) using "publickey".
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: filesystem
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: client_input_hostkeys: received ED25519 key SHA256:ybgmFkzwOSotHTHLJgHO0QN8L0xErw6vd0VhFA9m3SM
debug3: client_input_hostkeys: received ECDSA key SHA256:FC73VB6C4OQLSCrjEayhMp9UMxS97caD/Yyi2bhW/J0
debug3: client_input_hostkeys: received RSA key SHA256:46OSHA1Rmj8E8ERTC6xkNcmGOw9oFxYr0WF6zWW8l1E
debug1: client_input_hostkeys: searching /Users/isaasci/.ssh/known_hosts for bitbucket.org / (none)
debug3: hostkeys_foreach: reading file "/Users/isaasci/.ssh/known_hosts"
debug3: hostkeys_find: found ssh-ed25519 key at /Users/isaasci/.ssh/known_hosts:65
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key at /Users/isaasci/.ssh/known_hosts:66
debug3: hostkeys_find: found ssh-rsa key at /Users/isaasci/.ssh/known_hosts:67
debug1: client_input_hostkeys: searching /Users/isaasci/.ssh/known_hosts2 for bitbucket.org / (none)
debug1: client_input_hostkeys: hostkeys file /Users/isaasci/.ssh/known_hosts2 does not exist
debug3: client_input_hostkeys: 3 server keys: 0 new, 3 retained, 0 incomplete match. 0 to remove
debug1: client_input_hostkeys: no new or deprecated keys from server
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 5 setting TCP_NODELAY
debug3: set_sock_tos: set socket 5 IP_TOS 0x20
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env rvm_bin_path
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env ZDOTDIR
debug3: Ignored env ORIGINAL_XDG_CURRENT_DESKTOP
debug3: Ignored env MallocNanoZone
debug3: Ignored env GIT_REFLOG_ACTION
debug3: Ignored env USER
debug3: Ignored env COMMAND_MODE
debug3: Ignored env rvm_path
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env VIRTUAL_ENV
debug3: Ignored env rvm_prefix
debug3: Ignored env PATH
debug3: Ignored env _
debug3: Ignored env LaunchInstanceID
debug1: channel 0: setting env GIT_PROTOCOL = "version=2"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env __CFBundleIdentifier
debug3: Ignored env USER_ZDOTDIR
debug3: Ignored env PWD
debug3: Ignored env POETRY_ACTIVE
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env XPC_FLAGS
debug3: Ignored env VSCODE_GIT_ASKPASS_EXTRA_ARGS
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env rvm_version
debug3: Ignored env VSCODE_INJECTION
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env VSCODE_GIT_ASKPASS_MAIN
debug3: Ignored env GIT_SSH_COMMAND
debug3: Ignored env LOGNAME
debug3: Ignored env VSCODE_GIT_IPC_HANDLE
debug3: Ignored env VSCODE_GIT_ASKPASS_NODE
debug3: Ignored env GIT_ASKPASS
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env VIRTUAL_ENV_PROMPT
debug3: Ignored env GIT_EXEC_PATH
debug3: Ignored env COLORTERM
debug1: Sending command: git-upload-pack '<redacted>'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd ext data 165
The requested repository either does not exist or you do not have access. If you believe this repository exists and you have access, make sure you're authenticated.
debug2: channel 0: written 165 to efd 8
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: chan_shutdown_write: channel 0: (i0 o1 sock -1 wfd 7 efd 8 [write])
debug2: channel 0: output drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug2: chan_shutdown_read: channel 0: (i0 o3 sock -1 wfd 6 efd 8 [write])
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sfatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ock -1 cc -1 io 0x00/0x00)

debug3: send packet: type 1
Transferred: sent 3144, received 2576 bytes, in 0.3 seconds
Bytes per second: sent 9135.5, received 7485.1
debug1: Exit status 1

[Patrik S]

Hello @İsa aşcı and welcome to the Community!

From the SSH verbose logs, I see you have multiple keys in your local machine that will be tried in sequence to authenticate with Bitbucket : 

debug1: Will attempt key: /Users/isaasci/.ssh/id_rsa RSA SHA256:gxuV7n2lINJxAziDbTtbVWBg2FcJRC5wQ/ykPfgC78M agent
debug1: Will attempt key: /Users/isaasci/.ssh/id_ecdsa ECDSA SHA256:MItks7k4VMKSN7MbH70qDoSpoRgmmK/3Xee2aZ3G56c agent
debug1: Will attempt key: isaasci@192.168.1.35 ED25519 SHA256:MGgjWLrCQpfF/jakArgdNF4FAvuM8yNPgLc9JYFUOvw agent
debug1: Will attempt key: /Users/isaasci/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/isaasci/.ssh/id_ed25519
debug1: Will attempt key: /Users/isaasci/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/isaasci/.ssh/id_xmss
debug1: Will attempt key: /Users/isaasci/.ssh/id_dsa

The first key that authenticates successfully with Bitbucket will be used. However, this key doesn't necessarily have access to the repository you are trying to access. SSH keys added to your personal profile will have the same permissions to repos as your account has. Access keys, however, will have different levels of access depending on where they were added (repository, project, or workspace level) as described in Difference between repository, project and workspace access keys . 

So in this case, since you have multiple keys in your local environment, to make sure the correct key is being used to authenticate with Bitbucket, you can create a file named config (or edit the file if it already exists) under the folder  ~/.ssh/ with the following content : 

Host bitbucket.org
     HostName bitbucket.org
     User git
     IdentityFile ~/.ssh/<name of your ssh private key>
     IdentitiesOnly yes

This will instruct SSH to use the key in the path provided on IdentityFile option. If your key is in a different path please update the value of IdentityFile accordingly. Also, make sure the private key provided in the config file is the key to which the corresponding public key was added to your Bitbucket profile.

You can then confirm the correct key is being used by cloning the repository with verbose logs enabled : 

GIT_SSH_COMMAND="ssh -vvv" git clone git@bitbucket.org:<workspace>/<repository>.git   

 and look for the part of the logs where it prints : 

debug1: Offering public key: <path of your private key>

confirm if it matches with the path you have provided in the ~/.ssh/config file.

Hope that helps! Let us know in case you have any questions.

Thank you, @İsa aşcı !

Patrik S