Chaching submodules in bitbucket pipelines

[manukoch]

Hi there,

our repo has got several submodules. In order to reduce pipeline runtimes We'd like to cache the submodule data and fetch only the changes. 

Unfortunately the pipepline crashes if the submodules are cached because the initialiazation doesn't work proberly.

right now the following command gets executed

git submodule update --init

Does anyone of you know if submodule caching is possible and if so how to handle that.

[LanceRushing · Accepted solution]

The solution is to cache the .git/modules directory, and not the individual submodule directories.

pipelines:
  default:
    - step:
        caches:
          - git-modules
        script:
          - git submodule update --recursive --init
          - do-stuff

definitions:
  caches:
    git-modules: .git/modules

The "git module" command fails  b/c the directory exists (from cache) but there is no matching directory inside .git/modules.

Once I had it working, our "git submodule" step went from 35 seconds to 6 seconds.

[manukoch]

This works great, thank you! That's the solution I was looking for.

[manukoch]

The pipeline looks something similiar to this:

pipelines:
  default:
    - step:
        caches:
          - submodule1
          - submodule2
        script:
          - alter submodule1 url with pipeline user credentials
          - alter submodule2 url like the first one
          - "git submodule update --init"
          - run a gulp build
          - deploy
         
definitions:
  caches:
    submodule1: [submodule1 dir]
    submodule2: [submodule2 dir]

Without the caching the pipeline works fine. But with caching enabled the following error occurs:

fatal: Not a git repository: ../.git/modules/submodule1
Unable to find current revision in submodule path 'submodule1'

[Deleted user]

You should be able to cache your submodules the same way all other cache works.

In your bitbucket-pipelines.yml add a new definition of the submodule folder you would like to cache and then reference that cache in your build step.

Ref: Custom caches for other build tools and directories. 

Here is a small example. I have split the step of creating cache separate to the actual test as the cache is not built if the test fails. Though you can setup submodules and run tests in the same step.

pipelines:
  default:
    - step:
        name: Downloading Submodules
        caches:
          - submodules
        script:
          - git submodule update --init

definitions:
  caches:
    submodules: [submodule dir]

[manukoch]

Hi Wilhelm,

your solution is exactly what I've tried but unfortunately this doesn't work for me.

I always get this error: 

+ git submodule update
fatal: Not a git repository: ../.git/modules/NAME
Unable to find current revision in submodule path 'NAME' 

[Deleted user]

Hi @manukoch,

It looks like you missing the `--init` part of your `git submodule update` command. You should see the whole command in your history. 

[Deleted user]

Sorry for the double response, I thought I should look it up. Try wrap your git command in double quotes.

e.g. "git submodule update --init"

[manukoch]

Thanks for the advice but the double quotes aren't the solution for this. The pipeline still crashes.

[Deleted user]

Pleasure, if you would like to add part of your pipeline here I may be able to provide more assistance. 

[manukoch]

Sorry,  I  accidentally replyed to the thread instead to you.

[Raul Gomis]

Hi Manuel,

At the moment there is no support for git submodule caching in Pipelines. You could setup different pipelines in different repositories so that every repository clones its own changes. If that does not work for your particular use case, I recommend you open a ticket in our site master issue tracker (https://bitbucket.org/site/master/issues) with your use case so that we can track more customer uses cases and votes related to it.

Thanks,

Raul