git fetch does not work in bitbucket pipeline

[Naveen Joseph Gabriel]

While configuring bitbucket pipelines to deploy repo in google instance via ssh I noticed that git fetch does take lot of time or does not even complete. Does anyone know why it happens ?

Snippet of bitbucket command

COMMAND: >
         cd ../../services;
         if # something ; then   
         sudo git clone < clone link> &&
         cd <cloned directory >
         sudo git fetch -a &&
         sudo git checkout $BITBUCKET_BRANCH &&
         sudo docker compose up -d --build

Eventually I commented out fetch command and it still worked.

I figured out that during fetch it expects bitbucket password. 

Can someone tell me how to provide password ? I have tried URLS for fetch as well but still does not work. The only way I see is the delete the repo and recreate it each time.

[Theodora Boudale]

Hi @Naveen Joseph Gabriel,

I would like to ask for some additional information so I can better help you:

1. I noticed that you are not changing directories after the git clone command (in order to go into the clone's dir) and before you do a git fetch.

Is the directory named services a Git repo as well and is this the one you want to fetch references for? Or do you want to fetch references for the repo you cloned with the previous command?

2. Are these commands executed in the build container where your build is running, or on your Google instance after you connect to it from Pipelines?

Kind regards,
Theodora

[Naveen Joseph Gabriel]

Hello @Theodora Boudale 

Thanks for the response.

Point 1: There was a typo. I was indeed doing cd to the clone directory and after that tried doing fetch.

Is the directory named services a Git repo as well and is this the one you want to fetch references for? Or do you want to fetch references for the repo you cloned with the previous command? -- > services is a folder , within which I am cloning my repository with sudo git clone < clone link> . You are right in saying that I want to fetch references with the cloned repo.

Point 2 : Are these commands executed in the build container where your build is running, or on your Google instance after you connect to it from Pipelines?  - The commands are executed after ssh into google instance using pipe. The entire pipeline looks something like this :

- step :
  - script:
    - pipe: atlassian/ssh-run:0.4.1
      variables:
       SSH_USER: $SSH_USER
       SERVER: $TEST_SERVER_IP
       COMMAND: >
          < rest of the command script>

[Theodora Boudale]

Hi @Naveen Joseph Gabriel,

Thank you for the info.

If you clone the repo every time, you don't need to do a git fetch as well, because a clone will fetch all the references.

However, if you want to clone only once and then remove the git clone command from the script and do a git fetch only:

Since you are asked for a password, I assume that you use HTTPS to clone the repo?

If so, I would recommend setting up authentication with SSH keys instead, using an SSH key pair that doesn't have a passphrase. In this case, no password is needed as authentication is done with the keys.

If you only need to clone and pull from the repo (and not push to it), you could set up Access keys on the Google instance that provide read-only access.

You can see instructions here, depending on the OS of the Google instance:

• https://support.atlassian.com/bitbucket-cloud/docs/set-up-repository-access-keys-on-linux/
• https://support.atlassian.com/bitbucket-cloud/docs/set-up-repository-access-keys-on-windows/
• https://support.atlassian.com/bitbucket-cloud/docs/set-up-repository-access-keys-on-macos/

The docs have instructions to generate an SSH key pair (you need to do that on the Google instance) and then add the public key to the Bitbucket repo's Access keys (from Repository settings > Access keys). You could then use an SSH URL in the git clone command, and git fetch would also use the same SSH URL.

If you want to use HTTPS, is there a credential manager set up for Git on the Google instance?

You can figure that out if you connect to the Google instance as the user $SSH_USER (outside Pipelines) and run the command

git config --list

Is there any line with the text "credential.helper"? If so, which one is used?

Are there any saved credentials for bitbucket.org in that credential manager?

Kind regards,
Theodora

[Naveen Joseph Gabriel]

Hi @Theodora Boudale 

SSH keys were already set up and public key was kept in bitbucket repo.

So here is my doubt.

I had manually created the repo in the instance before I created the pipelines. So now if I I use fetch command from bitbucket pipeline it is not using ssh keys because I am assuming it ask for the password because execution of sudo git fetch -a does not gets completed.

Do you think manual creation of repo initially somehow prevents bitbucket pipelines to not use use ssh. If so, could you clarify what could be happening ?

Also, I had placed condition in command that if the directory already exist then don't run git clone command.  

Also could you reaffirm if deploying directly into instance using ssh is best practise ?

[Theodora Boudale]

Hi @Naveen Joseph Gabriel,

Manually creating the repo in the instance shouldn't be an issue.

1. If the repo now exists in the Google instance, can you navigate to the repo directory (outside Pipelines) and run the command

git remote -v

Does the URL in the output start with git@bitbucket.org or with https:// ? This is to confirm which type of authentication is used.

2. Can you post the full output of the git fetch command here (after sanitizing the actual repo name and workspace id)? If you use an SSH key pair without a passphrase, you shouldn't get asked for a password. I would like to see if the password prompt comes for bitbucket.org or if perhaps it's a prompt to give the password of the root user, since you are using sudo.

3. Is the $SSH_USER the root user or a different one?

In case they are different, which user did you use when you created the SSH keys in the Google instance? If you created the keys for $SSH_USER and $SSH_USER is not root, I don't believe that the SSH keys will be used, as with sudo the command will try to find SSH keys for the root user.

Also could you reaffirm if deploying directly into instance using ssh is best practise ?

I'm not sure if I understand the question. Do you mean if deploying via SSH is best instead of using HTTPS? Or if cloning the repo on the instance is ok?

If you do need a Git repo on this server, then what you're doing is ok. You need to be mindful though that the clone includes the repo's commits. It is possible to also deploy with rsync, scp, ftp.  If you don't want the .git folder on the Google instance, rsync, for example, will allow you to exclude it and copy only the source code files.

if you don't want the history of the repo but only the source code files; these tools will only copy the source code and not the Git repo.

It all depends on your requirements and it is not something I can give you a definitive answer for. If you have a systems architect in your company, I believe they would be the person to ask.

Kind regards,
Theodora

Edit: The info I posted earlier about rsync, scp, and ftp was incorrect, I just edited my response and corrected it to avoid confusion.

[Naveen Joseph Gabriel]

I figured out that during fetch it expects bitbucket password. 

Can someone tell me how to provide password ?