Using custom caches to build in pipelines

[Mats Alexander Nissen-Lie]

Hi,

I have a NX monorepo that I want to build and deploy to firebase in pipelines.

NX provides a build affected command that only builds the new changes made to the repo. However, I need the entire build compiled to deploy to firebase. Therefore, I am trying to cache the build in bitbucket. The build dist is not a part of the repo. How do I cache the folder between deploying, and can I have separate cache for building to dev and prod?

definitions:

caches:

my-bundler-cache:

 key:

  files:

    - dist

  path: dist

steps:

  - step: &deploy

   caches:

    - node

    - my-bundler-cache

 script:

  - npm ci

  - npm install -g firebase-tools

  - npx nx workspace-lint

  - npx nx format:check

 # - npx nx run-many --target=build --all --configuration="$NX_ENVIRONMENT" #used first time building

  - npx nx affected --target=lint --base=origin/master --parallel & npx nx affected --target=test --base=HEAD~1 --parallel --configuration="$NX_ENVIRONMENT" & npx nx affected --target=build --base=HEAD~1 --parallel

  - firebase deploy --only hosting --project="$NX_ENVIRONMENT" --token "$BITBUCKET_FIREBASE_TOKEN"

d

File structure:

[Patrik S · Accepted solution]

Hello @Mats Alexander Nissen-Lie  and welcome to Atlassian Community!

I think caching may not be efficient in this case. This is because when you configure a directory to be cached, like the dist directory in your case, the first time pipelines run after you have configured the cache, it will complete the build commands and in the Build teardown phase it will look at the dist folder, compress any file that is within that directory and upload it as a cache.

This cache will be available for all the subsequent builds for a week since they were generated, and they will not be automatically refreshed within that period unless a file you defined as a key file has changed (see Cache with file-based cache keys) or you have manually deleted the cache in the UI (Pipelines > Caches > Trash icon). Any cache which is older than 1 week will be cleared automatically and repopulated during the next build.

With that in mind, to achieve the use case you mentioned you would have to define a second pipeline - it could be a custom manually triggered pipeline - that runs the npx command that compiles the entire repo so that it would be used to "generate" the cache. Then you would have to trigger this second custom pipeline every 7 days to re-upload the cache, as it will be automatically cleared within a week of its creation. This is because if you run a build configured with the nx affected option when the cache is not populated, pipelines will generate the cache out of that build and, as you mentioned, it will only compile the changed files, meaning not the entire compiled repo will be under the dist folder and hence the cache will have only have that subset of files.

Hope that helps to clarify your questions!

Thank you, @Mats Alexander Nissen-Lie !

Patrik S