Chaching submodules in bitbucket pipelines

Hi there,

our repo has got several submodules. In order to reduce pipeline runtimes We'd like to cache the submodule data and fetch only the changes. 

Unfortunately the pipepline crashes if the submodules are cached because the initialiazation doesn't work proberly.

right now the following command gets executed

git submodule update --init

Does anyone of you know if submodule caching is possible and if so how to handle that.


4 answers

1 accepted

4 votes
Accepted answer

The solution is to cache the .git/modules directory, and not the individual submodule directories.

- step:
- git-modules
- git submodule update --recursive --init
- do-stuff

git-modules: .git/modules

The "git module" command fails  b/c the directory exists (from cache) but there is no matching directory inside .git/modules.

Once I had it working, our "git submodule" step went from 35 seconds to 6 seconds.

This works great, thank you! That's the solution I was looking for.

0 votes
Raul Gomis Atlassian Team Mar 28, 2018

Hi Manuel,

At the moment there is no support for git submodule caching in Pipelines. You could setup different pipelines in different repositories so that every repository clones its own changes. If that does not work for your particular use case, I recommend you open a ticket in our site master issue tracker ( with your use case so that we can track more customer uses cases and votes related to it.



You should be able to cache your submodules the same way all other cache works.

In your bitbucket-pipelines.yml add a new definition of the submodule folder you would like to cache and then reference that cache in your build step.

Ref: Custom caches for other build tools and directories

Here is a small example. I have split the step of creating cache separate to the actual test as the cache is not built if the test fails. Though you can setup submodules and run tests in the same step.

    - step:
name: Downloading Submodules caches: - submodules script: - git submodule update --init definitions: caches: submodules: [submodule dir]

Hi Wilhelm,

your solution is exactly what I've tried but unfortunately this doesn't work for me.

I always get this error: 

+ git submodule update
fatal: Not a git repository: ../.git/modules/NAME
Unable to find current revision in submodule path 'NAME' 

Hi @Manuel Koch,

It looks like you missing the `--init` part of your `git submodule update` command. You should see the whole command in your history. 

Sorry for the double response, I thought I should look it up. Try wrap your git command in double quotes.

e.g. "git submodule update --init"

Thanks for the advice but the double quotes aren't the solution for this. The pipeline still crashes.

Pleasure, if you would like to add part of your pipeline here I may be able to provide more assistance. 

Sorry,  I  accidentally replyed to the thread instead to you.

The pipeline looks something similiar to this:

    - step:
          - submodule1
- submodule2 script:
- alter submodule1 url with pipeline user credentials
- alter submodule2 url like the first one - "git submodule update --init"
- run a gulp build
- deploy definitions: caches: submodule1: [submodule1 dir]
submodule2: [submodule2 dir]

Without the caching the pipeline works fine. But with caching enabled the following error occurs:

fatal: Not a git repository: ../.git/modules/submodule1
Unable to find current revision in submodule path 'submodule1'

Suggest an answer

Log in or Sign up to answer
Community showcase
Published Nov 06, 2018 in Bitbucket

Upgrade Best Practices

Hello! My name is Mark Askew and I am a Premier Support Engineer for products Bitbucket Server/Data Center, Fisheye & Crucible. Today, I want to bring the discussion that Jennifer, Matt, and ...

657 views 5 9
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you