Can only push from some repositories (local to bitbucket)

cynthiabluteau January 6, 2022

This is my first post, so I apologize in advance if it lacks details. I'm struggling to push specific repositories to bitbucket.

Some repositories that were fine in pushing are still OK. Some old and new ones are not.

I created a few new repositories today, which I can successfully push to bitbucket. Some new repositories I cannot, though.  

I've tried creating new ssh-keys. When I checked my connection:

ssh -T git@bitbucket.org

I get

authenticated via ssh key.

You can use git to connect to Bitbucket. Shell access is disabled

 But using on a temperamental repository:

git push -u origin master

Yields: 

Forbidden

fatal: Could not read from remote repository.

 
Please make sure you have the correct access rights

I tried to see if it was an issue with a specific workspace or if the public/private repository made a difference. I found no common denominator 

Other repositories work as expected. I cannot find the problem -- it's too temperamental.

EDIT: 

It's two workspaces that are causing issues. 

3 answers

1 accepted

1 vote
Answer accepted
Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 10, 2022

Hi @cynthiabluteau,

I can see that you replied mentioning that you worked around this issue by pushing your code to another repo; if you like, we can still look into the issue with the other workspaces to find the root cause and a way for you to work with them.

If you'd like us to look into this, could you please let me know:

1. Do you have one Bitbucket Cloud account that has access to all these workspaces (the ones with the problem, and the ones without)?
Or do you perhaps have multiple Bitbucket accounts?

(In case you have multiple accounts, it's possible that the one you added your SSH keys to does not have access to one of the workspaces).

2. If you run the command git remote -v in the clone of a repo that has this issue, what is the format of the URL in the output?

Is it as follows?

git@bitbucket.org:<workspace-id>/<repo-slug>.git

3. Could you try making a push to one of the repos with this issue, with the following command, and then share the output here?

GIT_SSH_COMMAND="ssh -vvv" git push -u origin master

Please remove any sensitive/private info in the output (like workspace-id, repo name) prior to sharing it here.
This will give more verbose output and possibly an indication of what may be going wrong.

Kind regards,
Theodora

cynthiabluteau January 10, 2022

Answers below:

  1. I only have one account that has access to all three workspaces that I administer 
  2. Running the git remove -v command yields in both the problem workspace and those that work as expected
origin git@bitbucket.org:<workspace-id>/name_of_repo.git (fetch)
origin git@bitbucket.org:<workspace-id>/name_of_repo.git (push)

 

3. Running 

GIT_SSH_COMMAND="ssh -vvv" git push -u origin master

 yields a very long output in the next comment, and finds keys that are no longer in my ~/.ssh folder.




OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/myusername/.ssh/config
debug1: /Users/myusername/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to bitbucket.org port 22.
debug1: Connection established.
debug1: identity file /Users/myusername/.ssh/id_rsa type 0
debug1: identity file /Users/myusername/.ssh/id_rsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_dsa type -1
debug1: identity file /Users/myusername/.ssh/id_dsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_ecdsa type -1
debug1: identity file /Users/myusername/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_ed25519 type -1
debug1: identity file /Users/myusername/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/myusername/.ssh/id_xmss type -1
debug1: identity file /Users/myusername/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version conker_641a4464aa ceb68dfe53cb
debug1: no match: conker_641a4464aa ceb68dfe53cb
debug3: fd 8 is O_NONBLOCK
debug1: Authenticating to bitbucket.org:22 as 'git'
debug3: hostkeys_foreach: reading file "/Users/myusername/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/myusername/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-s
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug3: hostkeys_foreach: reading file "/Users/myusername/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/myusername/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: hostkeys_foreach: reading file "/Users/myusername/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/myusername/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys from 2406:da00:ff00::6b17:d1f5
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /Users/myusername/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/myusername/.ssh/id_rsa RSA SHA256:/omR6vtha43fQaMjaWwmR1/Qp7XO325Vdzucl4QJA8g
debug1: Will attempt key: /Users/myusername/.ssh/id_dsa
debug1: Will attempt key: /Users/myusername/.ssh/id_ecdsa
debug1: Will attempt key: /Users/myusername/.ssh/id_ed25519
debug1: Will attempt key: /Users/myusername/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ssh-rsa,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-512,ssh-dss>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/myusername/.ssh/id_rsa RSA SHA256:/omR6vtha43fQaMjaWwmR1/Qp7XO325Vdzucl4QJA8g
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/myusername/.ssh/id_rsa RSA SHA256:/omR6vtha43fQaMjaWwmR1/Qp7XO325Vdzucl4QJA8g
debug3: sign_and_send_pubkey: RSA SHA256:/omR6vtha43fQaMjaWwmR1/Qp7XO325Vdzucl4QJA8g
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: Search for item with query: {
acct = "/Users/myusername/.ssh/id_rsa";
agrp = "com.apple.ssh.passphrases";
class = genp;
labl = "SSH: /Users/myusername/.ssh/id_rsa";
nleg = 1;
"r_Data" = 1;
svce = OpenSSH;
"u_AuthUI" = "u_AuthUIF";
}
debug2: using passphrase from keychain
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to bitbucket.org ([2406:da00:ff00::6b17:d1f5]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 7 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 8 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x20
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env LIBRARY_PATH
debug3: Ignored env CONDA_SHLVL
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env CONDA_PROMPT_MODIFIER
debug3: Ignored env SDKROOT
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env USER
debug3: Ignored env CONDA_EXE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env CPATH
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env _CE_CONDA
debug3: Ignored env PATH
debug3: Ignored env _
debug3: Ignored env CONDA_PREFIX
debug3: Ignored env PWD
debug3: Ignored env EDITOR
debug1: Sending env LANG = en_AU.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env XPC_FLAGS
debug3: Ignored env _CE_M
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env GIT_SSH_COMMAND
debug3: Ignored env LOGNAME
debug3: Ignored env CONDA_PYTHON_EXE
debug3: Ignored env VISUAL
debug3: Ignored env CONDA_DEFAULT_ENV
debug3: Ignored env DISPLAY
debug3: Ignored env GIT_EXEC_PATH
debug1: Sending command: git-receive-pack 'workspace-id/repo-name.io.git'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd ext data 10
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug2: channel 0: chan_shutdown_read (i0 o1 sock -1 wfd 4 efd 9 [write])
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: obuf_empty delayed efd 9/(10)
Forbidden
debug2: channel 0: written 10 to efd 9
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 7 efd 9 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/9 sock -1 cc -1)

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
debug3: send packet: type 1
debug1: fd 0 clearing O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
Transferred: sent 3148, received 2196 bytes, in 0.1 seconds
Bytes per second: sent 22933.4, received 15998.0
debug1: Exit status 1

Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 11, 2022

Hi @cynthiabluteau,

Thank you for the info.

Looking at the verbose output of the push command, I can see that the following key is being offered and then accepted by Bitbucket:

/Users/myusername/.ssh/id_rsa

1. Is this the key that is added to your Bitbucket user account, from avatar > Personal settings > SSH keys?

You can compare with the public key you have locally if you run

cat /Users/myusername/.ssh/id_rsa.pub

2. Do you have multiple SSH key pairs in your ~/.ssh folder?

3. Is there a file named config in your ~/.ssh folder, and if so, does it have any configuration specific to bitbucket.org?

I suspect that there may be multiple SSH key pairs in ~/.ssh, and the key offered may not be the one associated with your Bitbucket user.

Kind regards,
Theodora

cynthiabluteau January 11, 2022

Thank you for your response.

 

1. id_rsa was indeed the key used by the bitbucket account, and used by the other repos until I created a new one just for bitbucket. This fixed one of my workspaces, but not the one posted above. Interestingly, Bitbucket no longer lets me add this key back after deleting it. The webpage says a user has already loaded this key (aka me I guess?). 

2. Yes, now I have multiple keys to fix my original problem.

3. Yes I do have a config file. Not sure what you mean specific to bitbucket. I used what I found on bitbucket website (I rechecked the support pages when I was troubleshooting last week):

Host *

UseKeychain yes

 Is there something else that should be in the config?

 

Some follow up questions

A. Why can I push from the command line to many repositories in my user profile but not this workspace?

B. If having multiple keys is the problem, why isn't it a problem with my repositories 2 other workspaces?

C. Is there a command to find out which keys are being used by the other repositories (maybe the same verbose output as you suggested above) 

Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 12, 2022

Hi @cynthiabluteau,

Please allow me to give some context first: SSH keys can be added in 3 different places in Bitbucket Cloud

  1. On your Bitbucket user (on https://bitbucket.org/ select your avatar > Personal settings > SSH keys).
    These keys will allow you to push to any repository your Bitbucket account has permissions to push.
  2. On a workspace (on https://bitbucket.org/ select your avatar > All workspaces > select a workspace to open it > select Settings for the workspace > SSH keys). This option is not available for your personal workspace, the one created automatically by Bitbucket when you sign up for an account. It is available for workspaces you created manually.
    These keys will allow you to push only to repositories of this specific workspace.
  3. On the repository level (if you open a repository on Bitbucket website > select Repository settings from the sidebar > Access keys)
    These keys will allow you to clone the repository you have added them to, but they won't allow you to push.

I checked your Bitbucket Cloud account in our system (the one with the same email as your community account) and I can see:
- there is an SSH key added to your user
- some of the workspaces you have access to also have an SSH key
- I am not able to see if any of your repos have Access keys

Based on the issue you are facing (you can push to repos of some workspaces, but not others), it sounds like the SSH key offered when you push may be one of the keys you have added to a certain workspace, and not to your user.


If this setup with multiple SSH keys was accidental and you don't have a specific need for that, I would recommend the following:

  • Remove the SSH keys from all your workspaces (on https://bitbucket.org/ select your avatar > All workspaces > select each workspace to open it > select Settings for the workspace > SSH keys)
  • Leave (do not remove) the SSH key that is added to your account (on https://bitbucket.org/ select your avatar > Personal settings > SSH keys)
  • Check the content of the public keys that you have locally and find the one that is added to your Bitbucket user.

You can then delete the rest of the keys locally (if you don't need them or use them elsewhere) and/or add the following to the config file of your ~/.ssh folder:

Host bitbucket.org
HostName bitbucket.org
User YourBitbucketUsername
PreferredAuthentications publickey
IdentityFile /Users/YourUser/.ssh/my_ssh_key

Replace YourBitbucketUsername with your Bitbucket username and /Users/YourUser/.ssh/my_ssh_key with the path to the private key whose public key is added to your Bitbucket user.

If the key used is the one you have added to your Bitbucket user, you should be able to push to all repos this user has at least write access to.


Regarding your questions A. and B., your ability to push depends on which SSH key is offered by your client when you are making a push (if it is a workspace key, you will only be allowed to push to that specific workspace).
Regarding C. I'm afraid that it is not possible to figure this out with a command.


Please feel free to let me know if you have any questions, if the solution I suggested works for you or if you have a different requirement.

Kind regards,
Theodora

cynthiabluteau January 12, 2022

Thank you,

I thought I had removed those keys. I could push to the repos with the workspace that contained its own  SSH-key (I have most of my things there, and didn't want to mess with that workspace since I needed to retain my ability to push for a project).

The problem workspace had its own ssh-key, which I deleted along with all other ssh-keys at the workspace level.

I also updated the config since one of the "obsolete" keys is used on my local network (I didn't want to delete it), but did delete the other workspace keys that I no longer need.

I can communicate with repos across all workspaces now with my user ssh-key.

Cheers,

Cynthia

Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 12, 2022

Hi Cynthia,

That's good to hear, thank you for the update!

You are very welcome, and please feel free to reach out if you ever need anything else.

Kind regards,
Theodora

0 votes
cynthiabluteau January 7, 2022

I deleted all my ssh-keys, and created a new key for my user. Two workspaces now work as expected, but the third does not. Tried setting the URL to my user name, no dice.

I deleted the workspace (there was only 1x repo), and managed to include the code into another repo... Not really the best solution but I can push changes now, and the code was pretty new so not many commits/changes.

0 votes
Pramodh M
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
January 6, 2022

Hi @cynthiabluteau 

Can you check the permissions associated with Repository or Project, this seems to be issue

Thanks,
Pramodh

cynthiabluteau January 7, 2022

I'm an admin on all repositories. 

cynthiabluteau January 7, 2022

Interestingly, I deleted my SSH-keys on my bitbucket account but the repositories that were working still do. I went to check if there were keys set for that repo, and there isn't either.

Pramodh M
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
January 7, 2022

This is the issue if you were using SSH for authentication

cynthiabluteau January 7, 2022

Not sure what you mean.

 

The SHH-keys are still stored on bitbucket since I could not re-add them after deleting them from everywhere (there was 1 repo that had ssh-keys, all other repos were working from those supplied at the user level).

 

The shh-keys were working for some repositories but not all. 

 

My answer is basically, I meddled around and managed to get something to work but not everything. 

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
AUG Leaders

Atlassian Community Events