Chaching submodules in bitbucket pipelines

Hi there,

our repo has got several submodules. In order to reduce pipeline runtimes We'd like to cache the submodule data and fetch only the changes. 

Unfortunately the pipepline crashes if the submodules are cached because the initialiazation doesn't work proberly.

right now the following command gets executed

git submodule update --init

Does anyone of you know if submodule caching is possible and if so how to handle that.

 

3 answers

0 vote
Raul Gomis Atlassian Team Mar 28, 2018

Hi Manuel,

At the moment there is no support for git submodule caching in Pipelines. You could setup different pipelines in different repositories so that every repository clones its own changes. If that does not work for your particular use case, I recommend you open a ticket in our site master issue tracker (https://bitbucket.org/site/master/issues) with your use case so that we can track more customer uses cases and votes related to it.

Thanks,

Raul

You should be able to cache your submodules the same way all other cache works.

In your bitbucket-pipelines.yml add a new definition of the submodule folder you would like to cache and then reference that cache in your build step.

Ref: Custom caches for other build tools and directories

Here is a small example. I have split the step of creating cache separate to the actual test as the cache is not built if the test fails. Though you can setup submodules and run tests in the same step.

pipelines:
  default:
    - step:
name: Downloading Submodules caches: - submodules script: - git submodule update --init definitions: caches: submodules: [submodule dir]

Hi Wilhelm,

your solution is exactly what I've tried but unfortunately this doesn't work for me.

I always get this error: 

+ git submodule update
fatal: Not a git repository: ../.git/modules/NAME
Unable to find current revision in submodule path 'NAME' 

Hi @Manuel Koch,

It looks like you missing the `--init` part of your `git submodule update` command. You should see the whole command in your history. 

Sorry for the double response, I thought I should look it up. Try wrap your git command in double quotes.

e.g. "git submodule update --init"

Thanks for the advice but the double quotes aren't the solution for this. The pipeline still crashes.

Pleasure, if you would like to add part of your pipeline here I may be able to provide more assistance. 

Sorry,  I  accidentally replyed to the thread instead to you.

The pipeline looks something similiar to this:

pipelines:
  default:
    - step:
        caches:
          - submodule1
- submodule2 script:
- alter submodule1 url with pipeline user credentials
- alter submodule2 url like the first one - "git submodule update --init"
- run a gulp build
- deploy definitions: caches: submodule1: [submodule1 dir]
submodule2: [submodule2 dir]

Without the caching the pipeline works fine. But with caching enabled the following error occurs:

fatal: Not a git repository: ../.git/modules/submodule1
Unable to find current revision in submodule path 'submodule1'

Suggest an answer

Log in or Sign up to answer
Atlassian Community Anniversary

Happy Anniversary, Atlassian Community!

This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.

Read more
Community showcase
Piotr Plewa
Published Dec 27, 2017 in Bitbucket

Recipe: Deploying AWS Lambda functions with Bitbucket Pipelines

Bitbucket Pipelines helps me manage and automate a number of serverless deployments to AWS Lambda and this is how I do it. I'm building Node.js Lambda functions using node-lambda&nbsp...

1,747 views 1 5
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you