Windows Runner Clone Submodules

[Rik Bruns]

Hello,

I have followed this guide to setup a self hosted Windows Runner. The runner is working however there is an issue when cloning submodules.  The pipeline fails due to permission issue.

git submodule update --init --recursive

///////////////////////////////

Host key verification failed.

fatal: Could not read from remote repository.

The SSH key section on the Windows Runner guide is a bit hard to follow.  It is unclear after step 5 where certain steps should happen (Remote Host, Virtual Machine, etc...).  I have generated a key set in PowerShell on the Windows machine.  I converted and copied the private key into a secret variable.  Trying to load the key in the example step results in a configuration error.

- [Text.Encoding]::ASCII.GetString([Convert]::FromBase64String($Env:MY_SSH_KEY)) > ./id_rsa - ssh -i ./id_rsa <user>@<host> 'echo "connected from pipelines"'

There is a YAML syntax error in your bitbucket-pipelines.yml at [line 11, column 28]. To be precise: expected <block end>, but found '<scalar>'

We have several other repos with submodules using Pipelines with Linux Docker images with no problems.  We set the Access Keys up as directed and all works well. 

Any guidance on how to clone the full repo (with submodules) on a Windows Runner would be helpful.

[Garrett_LoVerde]

@Theodora Boudale Is there a better way to do this yet on either Windows or Linux shell runners?

[Theodora Boudale]

Hi Garrett,

Authentication needs to be set up for the submodules in order for the runner to be able to clone different repos.

We have a feature request for Pipelines to clone and set up submodules as well during the Build setup:

• https://jira.atlassian.com/browse/BCLOUD-12853

Are you looking for the same in self-hosted runners?

Kind regards,
Theodora

[Garrett_LoVerde]

Yes. I would like that to work for self-hosted runners. But I also would need a lot of configurability for which submodules to clone depending on the pipeline.

[Theodora Boudale]

Hi Rik,

The error you mention in the beginning, Host key verification failed., means that the host key of bitbucket.org is not in the ~/.ssh/known_hosts file of the machine where you have the Windows runner.

You can run the following command in PowerShell:

ssh-keyscan -t rsa bitbucket.org

and then copy-paste the output in the ~/.ssh/known_hosts file of the Windows machine where you run the runner. If the file doesn't exist, you can create it.

Regarding the SSH keys, you said you have converted and copied the private key into a secret variable.
Make sure you add the public key to the submodule's Access keys.

The command from our documentation to decode the variable and create the private key file during the build is giving me an error as well, I'll need to check with our development team about this.

In the meantime, I managed to get around this error by including the command to convert in parentheses as follows

- ([Text.Encoding]::ASCII.GetString([Convert]::FromBase64String($Env:my_ssh_key))) | Out-File -Encoding "ASCII" id_rsa

I am also creating the private key file in a different way than the documentation's command, as that was giving me an error later on when cloning the submodule, about the key's format.

The following command is needed to make sure that Git will use the SSH key that was created during the build:

- $Env:GIT_SSH_COMMAND='ssh -i ./id_rsa -vvv'

To sum it all up, the following YAML file should work:

pipelines: 
  default:
    - step:
        runs-on:
          - self.hosted
          - windows
        script:
          - ([Text.Encoding]::ASCII.GetString([Convert]::FromBase64String($Env:my_ssh_key))) | Out-File -Encoding "ASCII" id_rsa
          - $Env:GIT_SSH_COMMAND='ssh -i ./id_rsa -vvv'
          - git submodule update --init --recursive

Please feel free to let me know how it goes and if you need further assistance.

Kind regards,
Theodora