Understanding cache in monorepos - lerna

[shx24ztu]

I've got a monorepo which uses Lerna to publish to NPM.

- root

  - packages/componentB

  - packages/componentA

  - packages/all-components

I want to avoid building every single component everytime and thought 

can I not store the packages/compoent/dist in cache

but when i set the path: 

image: node:20

definitions:

  caches:

    build-cache:

      key:

       files:

         - lerna.json

       path: packages/**/dist

..

pipelines:

  pull-requests:

    "**": 

       - step:

         name: "Build divergent"

         caches:

           - build-cache

         script: 

           - node_modules/.bin/lerna run build --since=origin/main

         artifacts:

           - packages/**/dist

the dist/* get sqashed into a single folder without the filestructure so Iit cannot access any package/ * / dist  files

if I download the artifacts this is what it looks like, the build builds the divergent but then errors as it cannot find the rest that is inside the ** folder



Am i doing it wrong?

[Theodora Boudale]

Hi @shx24ztu and welcome to the community!

You can do either of the following:

1. Use as path in the smart cache definition only packages

2. Define three separate smart caches in your yml file, one with path packages/componentA/dist, one with path packages/componentB/dist, and another one with path packages/all-components/dist. Please make sure to add all of them in your step's definition.

I will discuss with my team the behavior when using ** in the cache path.

Kind regards,
Theodora

[shx24ztu]

HI @Theodora Boudale ,

thanks for your answer

do you mean to create a cache for each monorepo item ie:

componentA:
  key:
    files:

      - packages/componentA/package.json


componentB:
  key:
    files:

      - packages/componentB/package.json
...etc

and reference the cache in each different step:

pipelines:
  pull-request:
    "**"
       - step:
          name: "Build changed component"
          caches:
             - componentA
             - componentB
             ...
       

pipelines:
  branches:
    "main"
       - step:
          name: "Publish"
          caches:
             - componentA
             - componentB
             ...
       

at the moment we have about a dozen components, but this will grow a lot, so this ads a bit of a strain to our DRY principle, do you have a way of populating these arrays in any clever way?

[Theodora Boudale]

Hi @shx24ztu ,

do you mean to create a cache for each monorepo item [...]

Yes, this is what I meant with the suggestion suggestion.

do you have a way of populating these arrays in any clever way?

I don't see a way to do that at the moment, but I'm still waiting for feedbck from my team in case I am missing something. Please allow me some more time, and I will get back to you on this.

Kind regards,
Theodora

[Theodora Boudale]

Hi @shx24ztu,

I'm afraid the directory structure cannot be preserved if you use a single cache definition with a wildcard in the path.

I went ahead and created a request to preserve the dir structure in our issue tracker:

• https://jira.atlassian.com/browse/BCLOUD-23246

I would suggest adding your vote to that request (by selecting the Vote for this issue link) to express your interest. You are more than welcome to leave feedback, and you can also add yourself as a watcher (by selecting the Start watching this issue link) if you'd like to get notified via email on updates.

Implementation of features is done as per our policy here and any updates will be posted in the feature request.

In the meantime, you can use multiple cache definitions, one for each directory.

Kind regards,
Theodora