Git submodules on Bitbucket pipelines over https

[Javier Fernández Pastrana]

Hi,

As we well known, the support of Git over https is ended and we are facing problems in our Bitbucket pipeline.

We have configured three submodules in our main repository and we have configured these submodules over https because we cannot use ssh (as in many companies happens). Locally is fine, because we have configure the APP Pasword, and works well.

However, I do not find a way to configure the same on the pipeline, it returns the following error:

CURRENTLY:

The configuration that I have is:

BEFORE:

What we had before, and it is not longer working is:

It was a workaround, to reconfigure the submodule to use Git over ssh.

For that solution also we had configured ssh keys between repositories to make them accesible.

I have tried many different approaches without success.... any suggestion?

Best regards,

Javier

[Luis Plaza]

Hi @Javier Fernández Pastrana

the use of Bitbucket downloads to centralize dependencies instead of git modules can be another option that your team should take into account.

https://bitbucket.org/<workspace>/<repo>/downloads/tab=branches

You may store them using a pipe in the module repo and download and decompress of the artifact in the step script of the dependent repo.

Git submodules may look to make your life easier like symlinks but it is not recommended.

Kind regards,

Luis Plaza 

[Theodora Boudale]

Hi Javier,

I haven't been able to reproduce this error when using the configuration you mentioned in your answer.

Are you using a public Docker image as a build environment for your builds?
If so, could you please let me know which Docker image you are using for this step?

I would like to try and reproduce this with the Docker image you are using, to see if the issue may be specific to the image or the git version installed in the image.

Kind regards,
Theodora

[Javier Fernández Pastrana]

Hi @Theodora Boudale ,

I am using "ubuntu:hirsute" public image and the installed git version is 2.30.2.

Also I would like to add that I am using private and public repositories in the submodules. Two of them are private, from Bitbucket too and the other is public from Github.

I do not have any preference at all with the Docker image, could you tell me which one are you using? and I will try.

Thank you very much for the support,

Javier

[Theodora Boudale]

Hi @Javier Fernández Pastrana,

Thank you for the info. It doesn't look that the Docker image is the problem here, I have no issues updating the submodules when I use the Docker image you mentioned.

I have been able to reproduce this error though when the name used for the submodule in the following command (the my-submodule part)

git config submodule.my-submodule.url https://${BB_username}:${BB_password}@bitbucket.org/<workspace-id>/<repo>.git

does not match the name defined in the .gitmodules file of the repo. In this case, the url of the submodule is not changed, and the git submodule update command is using the url defined in .gitmodules (which doesn't include a username and app password)

• Does the submodule repo perhaps have any capital letters in its name?
• If you open the .gitmodules file for the repo, do the submodule name and path have any capital letters or are they lowercase?
• In the git config command you are using in your yml file, does the my-submodule part include any capital letters?

Kind regards,
Theodora

[Javier Fernández Pastrana]

Hi @Theodora Boudale ,

First of all, thank you very much for your effort.

Secondly, in git submodules is not specified the username and password, you can find it below:

Therefore, it has sense the first error when we run "git submodule update --init --recursive" that could not read Username because it is not specified....

Meanwhile, the second option that I tried I think that it should work.

Regarding your questions, I always use lower case for all the repositories. Other thing that may is important is the following:

Main repository - belongs to PROJECT A

• Git submodule 1 - Belongs to project B
• Git submodule 2 - Belongs to project B
• Git submodule 3 - Belongs to Github

I was doing some more test using my user and password instead of the the APP Paswords, but without sucess, I share the outputs of .git/config

FIRST "cat .git/config"

SECOND "cat .git/config"

And it returns the same error.....

thank you very much,

Javier

[Javier Fernández Pastrana]

Hi,

UPDATE

I am doing some tests in new and empty repositories and now I know the reason but not how to solve yet.

I will describe my scenario:

Main repository: 

• git submodule 1
• git submodule 2 "PROTO" folder
• git submodule 3 (github) "DEV_TOOLS/protobuf-c" folder

Therefore the hierarchy of the repository would be:

/

• .git
• README.md
• DEV_TOOLS
  • protobuf-c
    • .git
    • README.md
• PROTO
  • .git
  • README.md

If I remove this hierarchy and just put all the submodules into the root directory, it is working using git config and Bitbucket APP password!

Thus, we have to look for a way to configure the submodules inside of sub-directories inside of "yml" file.

Sorry to not specify this configuration from the beginning.

Hope it helps,

Javi

[Javier Fernández Pastrana]

[POSSIBLE SOLUTION]

I have found a possible solution to clone git submodules over https inside of sub-directories.

Instead of using "git config submodule.NAME.url <PATH>", I used "git submodule set-url -- <PATH> <URL>"

A basic pipeline will look as:

image: ubuntu:hirsute

clone:

  depth: full    # SonarCloud scanner needs the full history to assign issues properly

definitions:

  steps:

    - step: &StaticAnalysis

        name: Run unit tests

        script:

          - apt-get update

          - DEBIAN_FRONTEND="noninteractive" apt-get -y install git clang-format-12        

          - git config submodule.test-repo-1-fw-prj.url https://${BB_username}:${BB_password}@bitbucket.org/<ORGANIZATION>/test-repo-1-fw-prj.git

          - git submodule set-url -- PROTO https://${BB_username}:${BB_password}@bitbucket.org/<ORGANIZATION>/test-repo-2-fw-prj.git

          - git submodule set-url -- DEV_TOOLS/protobuf-c https://github.com/protobuf-c/protobuf-c.git

          - git submodule update --init --recursive --depth=1

pipelines:

  pull-requests:

    '**': #this runs as default for any branch not elsewhere defined

      - step: *StaticAnalysis

  branches:

    develop:

      - step: *StaticAnalysis

    master:

      - step: *StaticAnalysis

@Theodora Boudale could you confirm this solution? Because, I am not sure that it is going to checkout the proper commit of each submodule. Or it is going to checkout just the default branch for each submodule.

Many thanks again, this community is great,

Javi

[Theodora Boudale]

Hi Javi,

Thank you for sharing all that info.

If the configuration in the .gitmodules is as follows:

[submodule "REPO"]
        path = REPO
        url = https://bitbucket.org/my-workspace/test-repo.git

Then the command to update the URL in the yml file should be as follows:

git config submodule.REPO.url https://${BB_username}:${BB_password}@bitbucket.org/my-workspace/test-repo.git

I believe that the clone of the submodule may have been failing because you were using e.g. test-repo instead of REPO in the above command.

The other command you mentioned should also work:

git submodule set-url -- REPO https://${BB_username}:${BB_password}@bitbucket.org/my-workspace/test-repo.git

The following command:

git submodule update --init --recursive --depth=1

will clone only the commit of the submodule where the main repo is pointing to (by main repo I mean the repo that uses the submodule). This is the commit displayed on the Source page of the main repo, in the Bitbucket website.

If you have made any changes in the submodule, but you haven't updated the main repo to track the updated version of the submodule, then the submodule commit that will be cloned will not be the latest one (it will be the one displayed on the Source page of the main repo).

What is your requirement here? I'm not sure you mean by proper commit.
Do you want your submodule to point to a branch other than its main one?

Kind regards,
Theodora

[Javier Fernández Pastrana]

Hi @Theodora Boudale ,

Many thanks for your complete explanation. I think that key was:

git config submodule.REPO.url https://${BB_username}:${BB_password}@bitbucket.org/my-workspace/test-repo.git

As you said, I was using the test-repo in the URL section. My mistake. 

I was not completely sure about cloning the main repository which its specific references of each submodule. I will explain it with an exaplame:

* Main repository: Pull Request branch and pointing to:

• Submodule 1: develop branch
• Submodule 2: release/v1.7
• Submodule 3 : release/v2.0

I was not sure what was happening when is cloning the repository if it was cloning the specific reference of each or the default ones.

Any way, with your explanation now it is much more clear and the pipeline is doing what we expect to do.

Many thanks again, great support.

Regards,

Javi

[Theodora Boudale]

Hi Javi,

You are very welcome, I'm glad to have helped!

Regarding the other question with the branches:

If you have in the main repo a branch named e.g. feature1, navigate to the Source page of the repo on Bitbucket website, switch the branch from the dropdown to feature1 branch in order to see feature1's Source. The commit that is displayed there for each submodule is the one that will get cloned.

If you want the branch e.g. feature1 of the main repository to point to a specific branch of the submodule, e.g. branch release/v1.7, then you need to do the following:

1. Check out locally feature1 branch of the main repo

2. Adjust the .gitmodules file in the branch to look as follows:

[submodule "REPO"]
        path = REPO
        url = https://bitbucket.org/my-workspace/test-repo.git
        branch = release/v1.7

3. Then, run git submodule update --remote REPO

4. Commit your changes and push to the main repo.

This will update the submodule object in feature1 branch of the main repo, to point to the latest commit of release/v1.7

Perhaps you have followed this process already and it works as expected for you? I'm not sure. I just thought I would share the steps just in case.

Kind regards,
Theodora

[Javier Fernández Pastrana]

Extra information:

I have also tried the following approach How to use git submodule init property? without success: