How to make Bamboo work with Git submodules?

[Slavomir Danas]

I simply cannot get Bamboo and Bitbucket to work together in a superproject with submodules. Bamboo build fails when trying to checkout a Bitbucket repository during the submodules phase with authentication error (publickey).

I was googling for 2 hours to came to hint that one must set the Remote Repository type to Git not Bitbucket in Bamboo Plan properties, however this changed nothing.

I did set-up pubkey authentication, imported keys into Bamboo, added keys to Bitbucket project, cleared Bamboo repository cache, ran the build and got error.

Looking at the full log always yields:

checkout of the superproject succeeds
...
17-Jan-2020 11:17:39 remote: Compressing objects: 100% (48/48), done. 
17-Jan-2020 11:17:39 remote: Total 48 (delta 19), reused 0 (delta 0) 
17-Jan-2020 11:17:39 From ssh://127.0.0.1:43957/tis/ebe_tis
17-Jan-2020 11:17:39 * [new branch] master -> master
17-Jan-2020 11:17:39 /usr/bin/git log -1 --encoding=UTF-8 --format=%H 41cf8bc80f5accd26c5f506e9b25006523df67e0
17-Jan-2020 11:17:39 Checking out revision 41cf8bc80f5accd26c5f506e9b25006523df67e0.
17-Jan-2020 11:17:39 Creating local git repository in '/var/atlassian/application-data/bamboo/xml-data/build-dir/360449/MTIS-BET-JOB1/.git'.
17-Jan-2020 11:17:39 /usr/bin/git clone /var/atlassian/application-data/bamboo/xml-data/build-dir/_git-repositories-cache/5d231e8c912b82fa56f8043fc661bd70ac857a92 --shared --no-checkout '-c filter.lfs.smudge=' '-c filter.lfs.required=false' /var/atlassian/application-data/bamboo/xml-data/build-dir/360449/MTIS-BET-JOB1
17-Jan-2020 11:17:39 Cloning into '/var/atlassian/application-data/bamboo/xml-data/build-dir/360449/MTIS-BET-JOB1'...
17-Jan-2020 11:17:39 done.
17-Jan-2020 11:17:39 /usr/bin/git show-ref master
17-Jan-2020 11:17:39 /usr/bin/git checkout -f 41cf8bc80f5accd26c5f506e9b25006523df67e0
17-Jan-2020 11:17:39 Note: checking out '41cf8bc80f5accd26c5f506e9b25006523df67e0'.
17-Jan-2020 11:17:39 
17-Jan-2020 11:17:39 You are in 'detached HEAD' state. You can look around, make experimental
17-Jan-2020 11:17:39 changes and commit them, and you can discard any commits you make in this
17-Jan-2020 11:17:39 state without impacting any branches by performing another checkout.
17-Jan-2020 11:17:39 
17-Jan-2020 11:17:39 If you want to create a new branch to retain commits you create, you may
17-Jan-2020 11:17:39 do so (now or later) by using -b with the checkout command again. Example:
17-Jan-2020 11:17:39 
17-Jan-2020 11:17:39 git checkout -b <new-branch-name>
17-Jan-2020 11:17:39 
17-Jan-2020 11:17:39 HEAD is now at 41cf8bc Merge branch 'master' of https://git.mygitserver.com/scm/tis/ebe_tis
17-Jan-2020 11:17:39 /usr/bin/git remote set-url origin ssh://6587122c-5f97-4900-a3c9-69723488bee9@127.0.0.1:43957/tis/ebe_tis.git
17-Jan-2020 11:17:39 /usr/bin/git submodule update --init --recursive
17-Jan-2020 11:17:39 Submodule 'mqtt_api' (ssh://git@git.mygitserver.com:7999/tis/mqtt_api.git) registered for path 'mqtt_api'
17-Jan-2020 11:17:39 Submodule 'mqtt_calc' (ssh://git@git.mygitserver.com:7999/tis/mqtt_calc.git) registered for path 'mqtt_calc'
17-Jan-2020 11:17:39 Submodule 'mqtt_client' (ssh://git@git.mygitserver.com:7999/tis/mqtt_client.git) registered for path 'mqtt_client'
17-Jan-2020 11:17:39 Submodule 'mqtt_data' (ssh://git@git.mygitserver.com:7999/tis/mqtt_data.git) registered for path 'mqtt_data'
17-Jan-2020 11:17:39 Submodule 'mqtt_server' (ssh://git@git.mygitserver.com:7999/tis/mqtt_server.git) registered for path 'mqtt_server'
17-Jan-2020 11:17:39 Submodule 'mqtt_tecGUI' (ssh://git@git.mygitserver.com:7999/tis/mqtt_tecGUI.git) registered for path 'mqtt_tecGUI'
17-Jan-2020 11:17:39 Submodule 'mqtt_tests' (ssh://git@git.mygitserver.com:7999/tis/mqtt_tests.git) registered for path 'mqtt_tests'
17-Jan-2020 11:17:39 Submodule 'mqtt_utils' (ssh://git@git.mygitserver.com:7999/tis/mqtt_utils.git) registered for path 'mqtt_utils'
17-Jan-2020 11:17:39 Cloning into '/var/atlassian/application-data/bamboo/xml-data/build-dir/360449/MTIS-BET-JOB1/mqtt_api'...
17-Jan-2020 11:17:39 Warning: Permanently added '[git.mygitserver.com]:7999,[123.45.67.890]:7999' (RSA) to the list of known hosts.
17-Jan-2020 11:17:39 git@git.mygitserver.com: Permission denied (publickey).
17-Jan-2020 11:17:39 fatal: Could not read from remote repository.
17-Jan-2020 11:17:39 
17-Jan-2020 11:17:39 Please make sure you have the correct access rights
17-Jan-2020 11:17:39 and the repository exists.
17-Jan-2020 11:17:39 fatal: clone of 'ssh://git@git.mygitserver.com:7999/tis/mqtt_api.git' into submodule path '/var/atlassian/application-data/bamboo/xml-data/build-dir/360449/MTIS-BET-JOB1/mqtt_api' failed
17-Jan-2020 11:17:39 Failed to clone 'mqtt_api'. Retry scheduled

 Funny thing is going to Bamboo server, su to bamboo user do the same and it succeeds.. I'm lost...

root@jira-conf-bitb ~ # su bamboo
bamboo@jira-conf-bitb:~$ cd ~/temp
bamboo@jira-conf-bitb:~/temp$ which git
/usr/bin/git
bamboo@jira-conf-bitb:~/temp$ git clone ssh://git@git.mygitserver.com:7999/tis/ebe_tis.git
Cloning into 'ebe_tis'...
The authenticity of host '[git.mygitserver.com]:7999 ([123.45.67.890]:7999)' can't be established.
RSA key fingerprint is SHA256:ERYJFEDEfjgFYYHuznEnLvPAxv1lj2sbgIg+obvjx6Y.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[git.mygitserver.com]:7999,[123.45.67.890]:7999' (RSA) to the list of known hosts.
remote: Enumerating objects: 67, done.
remote: Counting objects: 100% (67/67), done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 67 (delta 30), reused 0 (delta 0)
Receiving objects: 100% (67/67), 11.41 KiB | 3.80 MiB/s, done.
Resolving deltas: 100% (30/30), done.
bamboo@jira-conf-bitb:~/temp$ cd ebe_tis/
bamboo@jira-conf-bitb:~/temp/ebe_tis$ git submodule update --init --recursive
Submodule 'mqtt_api' (ssh://git@git.mygitserver.com:7999/tis/mqtt_api.git) registered for path 'mqtt_api'
Submodule 'mqtt_calc' (ssh://git@git.mygitserver.com:7999/tis/mqtt_calc.git) registered for path 'mqtt_calc'
Submodule 'mqtt_client' (ssh://git@git.mygitserver.com:7999/tis/mqtt_client.git) registered for path 'mqtt_client'
Submodule 'mqtt_data' (ssh://git@git.mygitserver.com:7999/tis/mqtt_data.git) registered for path 'mqtt_data'
Submodule 'mqtt_server' (ssh://git@git.mygitserver.com:7999/tis/mqtt_server.git) registered for path 'mqtt_server'
Submodule 'mqtt_tecGUI' (ssh://git@git.mygitserver.com:7999/tis/mqtt_tecGUI.git) registered for path 'mqtt_tecGUI'
Submodule 'mqtt_tests' (ssh://git@git.mygitserver.com:7999/tis/mqtt_tests.git) registered for path 'mqtt_tests'
Submodule 'mqtt_utils' (ssh://git@git.mygitserver.com:7999/tis/mqtt_utils.git) registered for path 'mqtt_utils'
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_api'...
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_calc'...
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_client'...
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_data'...
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_server'...
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_tecGUI'...
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_tests'...
Cloning into '/var/atlassian/application-data/bamboo/temp/ebe_tis/mqtt_utils'...
Submodule path 'mqtt_api': checked out '6ac033a3854760748f0c2296110615b548720261'
Submodule path 'mqtt_calc': checked out '7e678e9030868b38da2aabafd38521bb0ce4adc7'
Submodule path 'mqtt_client': checked out '02b7374a22598b4d01f2aff132d7570e5fd01469'
Submodule path 'mqtt_data': checked out 'e35bc71d9fef07550de0bf615dedf54d8738bfa6'
Submodule path 'mqtt_server': checked out '4c6db73677a5c93a8408b70ae893f8b7e5666a32'
Submodule path 'mqtt_tecGUI': checked out '0cbb899c8c743326725218c543b30b9fd476d464'
Submodule path 'mqtt_tests': checked out '1a8458dcc19b2c58d6cca9d8dc106dd51775198a'
Submodule path 'mqtt_utils': checked out '411ce1825115f4beb69c01c9a51b6c14c532e3bb'
bamboo@jira-conf-bitb:~/temp/ebe_tis$ 

[Minh Tran]

@Slavomir Danas It seems the user running the job in Bamboo instance does not have permission to clone the git repo

Can you double check if the "bamboo" user is used to run in Bamboo instance?