nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Chaching submodules in bitbucket pipelines

manukoch

Hi there,

our repo has got several submodules. In order to reduce pipeline runtimes We'd like to cache the submodule data and fetch only the changes.

Unfortunately the pipepline crashes if the submodules are cached because the initialiazation doesn't work proberly.

right now the following command gets executed

git submodule update --init

Does anyone of you know if submodule caching is possible and if so how to handle that.

Accepted answers

LanceRushing

The solution is to cache the .git/modules directory, and not the individual submodule directories.

pipelines:
  default:
    - step:
        caches:
          - git-modules
script:
          - git submodule update --recursive --init
          - do-stuff

definitions:
  caches:
    git-modules: .git/modules

The "git module" command fails b/c the directory exists (from cache) but there is no matching directory inside .git/modules.

Once I had it working, our "git submodule" step went from 35 seconds to 6 seconds.

All comments

Raul Gomis

Hi Manuel,

At the moment there is no support for git submodule caching in Pipelines. You could setup different pipelines in different repositories so that every repository clones its own changes. If that does not work for your particular use case, I recommend you open a ticket in our site master issue tracker (https://bitbucket.org/site/master/issues) with your use case so that we can track more customer uses cases and votes related to it.

Thanks,

Raul

unknown

You should be able to cache your submodules the same way all other cache works.

In your bitbucket-pipelines.yml add a new definition of the submodule folder you would like to cache and then reference that cache in your build step.

Ref: Custom caches for other build tools and directories.

Here is a small example. I have split the step of creating cache separate to the actual test as the cache is not built if the test fails. Though you can setup submodules and run tests in the same step.

pipelines:
  default:
    - step:<br>        <span>name</span><span>: Downloading Submodules</span>
        caches:
          - submodules
        script:
          - git submodule update --init

definitions:
  caches:
    submodules: [submodule dir]

manukoch

Hi Wilhelm,

your solution is exactly what I've tried but unfortunately this doesn't work for me.

I always get this error:

+ git submodule update
fatal: Not a git repository: ../.git/modules/NAME
Unable to find current revision in submodule path 'NAME'

unknown

Hi @manukoch,

It looks like you missing the `--init` part of your `git submodule update` command. You should see the whole command in your history.

unknown

Sorry for the double response, I thought I should look it up. Try wrap your git command in double quotes.

e.g. "git submodule update --init"

manukoch

The pipeline looks something similiar to this:

<span>pipelines:</span>
  default:
    - step:
        caches:
          - submodule1<br>          - submodule2
        script:<br>          - alter submodule1 url with pipeline user credentials<br>          - alter submodule2 url like the first one
          - "git submodule update --init"<br>          - run a gulp build<br>          - deploy
         
<span>definitions:</span>
  caches:
    submodule1: [submodule1 dir]<br>    submodule2: [submodule2 dir]

Without the caching the pipeline works fine. But with caching enabled the following error occurs:

<span>Unable to find current revision in submodule path 'submodule1'</span><span>fatal: Not a git repository: ../.git/modules/submodule1<br></span>

LanceRushing

The solution is to cache the .git/modules directory, and not the individual submodule directories.

pipelines:
  default:
    - step:
        caches:
          - git-modules
script:
          - git submodule update --recursive --init
          - do-stuff

definitions:
  caches:
    git-modules: .git/modules

The "git module" command fails b/c the directory exists (from cache) but there is no matching directory inside .git/modules.

Once I had it working, our "git submodule" step went from 35 seconds to 6 seconds.

manukoch

This works great, thank you! That's the solution I was looking for.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bitbucket

bitbucket-pipelines