Is Yarn supported for bitbucket-pipelines caching?

[Shane Fast]

I've been using the node cache feature for some time now and have been using yarn to load my load modules with the following pipeline snippet:

pipelines: 
  default: 
    - step: 
      caches: 
        - node 
      script: 
        - yarn install
.
.
.

Observing the performance over several pipelines I haven't noticed any reduction in load time. I'm aware that the cache may not trigger for every pipeline, but I wanted to know if using yarn instead of npm is currently supported or is planned to be supported by pipelines caching.

[Ana Retamal · Accepted solution]

Hi Shane, can you confirm if you're managing your dependencies correctly as explained at Manage dependencies with Yarn.

Cheers,

Ana 

[Emilio Martinez]

I can't reply for Shane, but I am following the same process outlined in the doc you linked to, but am not seeing any benefits either. I believe I used to before but it's been a while since I noticed no benefits. At this time, my `yarn install`s take over half a minute despite the cache.

Incidentally, at the bottom I see `Cache "node": Skipping upload for previously downloaded cache`

[Deleted user]

Once install.sh is downloaded and executed, it will check if Yarn is already installed, if it does it'll just continue to the next script. FYI the script has 6.71 KB in size, so there's not too much to download :)

Cheers.

[Dan Rumney]

I think these answers are missing the point.

This isn't about Bitbucket caching the yarn installation... it's about caching the modules that yarn installs... the dependencies of the project being built.

I routinely see my `yarn install`s taking ~3m... they should take less than a second if my node_modules directory was being cached properly

[Shane Fast]

I did swap to npm install just for pipeline builds for the reduction in build times, but still use yarn install for my dev environment/production. 

[Deleted user]

After playing around a bit, I've found that the following currently works

pipelines:
  default:
    - step:
        name: Build
        image: node:6.11.4
        caches:
          - yarncustom
        script:
          - curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.3.2
          - export PATH=$HOME/.yarn/bin:$PATH
          - yarn
          - yarn build
definitions:
  caches:
    yarncustom: /usr/local/share/.cache/yarn/v1

I got here by executing "yarn cache dir" on a Bitbucket Pipeline and then just setup a custom cache as per the documentation. I then executed my build subsequent times with "yarn --verbose". I was able to confirm that builds were using the cache by seeing if it needed to download the modules again.

[Mike_Stead]

@[deleted] Thanks that worked for me too and double checked like you did. The cache download time is almost as slow as pulling from yarn but still a little boost.

Another point, the node docker image contains yarn so don't think there's a reason to install it via curl.. unless I'm missing something.

[Alex Bergin]

Yeah, in pipelines, I've found yarn sadly very slow, even with the cache workaround from @[deleted], and have switched back to npm install. Thanks for trying to help all.

[sondrel]

I was just about to try this out myself, but yesterday the `yarn install` step started executing in 1 second.

Seams like they might have fixed the caching in the pipelines, can anyone else confirm this?

[nakul_predixt]

Using 'yarn' to install all dependencies used to take around 6mins a couple of months ago -> today it only took 40s. After that it took 1s so it seems like the caching is working. 

[Oscar Origen]

Hi @nakul_predixt and @sondrel could you share your config file. My yarn command still take like 60-90s to execute.

[Oscar Origen]

The normal `-node` cache doesn't work with yarn and triggers the whole installation. 

However a custome cache: ` nodecustom: ./node_modules` works well with yarn and it just takes about 1s. 

[sondrel]

@Oscar OrigenDefault pipeline below:

image: node:11

pipelines:
  default:
    - step:
      caches:
        - node
      script:
        - yarn
        - yarn test
        - yarn build

[Arthur Simon]

`yarncustom: ./node_modules` solution worked for me.

But the first time you run it will be slow because it will be building the cache.

Subsequent deployments will be quick.

[Attila Szeremi]

Here's what I needed to do in December 2019 to get Yarn to take as fast as <1s.

image: node:10.17.0

pipelines:
  default:
    - step:
        name: Packages
        caches:
          - nodecustom
          - yarn
        script:
          - yarn

definitions:
  caches:
    nodecustom: ./node_modules
    yarn: /usr/local/share/.cache/yarn

Yes, nodecustom is necessary as per @Oscar Origen . Just "node" is not good enough. Don't ask why.

For yarn caching, you don't include the /v1 bit. At this time /v6 is being used; it's simpler if we just cache the version's parent directory to future-proof.

2023 October edit:

Now that Bitbucket Pipelines has file-based cache key support, the way to get fastest CI runs on average (1s for the yarn install part most of the time) is to use file-based caching for node_modules:
 

image: node:10.17.0

pipelines:
  default:
    - step:
        name: Packages
        caches:
          - nodecustom
        script:
          - yarn

definitions:
  caches:
    nodecustom:
      key:
        files:
          - yarn.lock
      path: ./node_modules

This way even if you're working on different branches where yarn.lock is different, it'll still only install the dependencies for real 1x per yarn.lock hash, and yarn install will take 1s from then on on both branches without the situation where only one branch gets 1s installation times and the other requires a proper install. (up to the cache expiry of 1 week of course).

Also, notice how I'm not using the yarn shared cache anymore. This is intentional. I don't recommend using it at all, because using it adds ~30 seconds your CI time for loading that data for every run unnecessarily, even if yarn install would take 1s either way. Sure, fresh installs would become faster with the yarn shared cache, but 95% of the time you'll just be hitting the nodecustom cache and get 1s yarn install times.

[Justin Hamade]

Thanks @Attila Szeremi This should be the accepted answer.

[Justin McMahon]

@Attila Szeremi's solution appears to work unless you're using yarn workspaces. I've spent almost two work days trying to figure out how to cache dependencies when using yarn workspaces and have come up with nothing. I'd appreciate any ideas.

Also, for what it's worth, Bitbucket Pipelines takes ~13mins to run all my steps whereas AWS CodeBuild runs the same pipeline with no cache in under 5mins.

[chason]

I have the same troubles as @Justin McMahon i'm using yarn workspace and yarn install is still 1min not 1s.

[viljar.kargenberg]

I managed to get the yarn to run in 1s when using yarn workspaces by creating additional caches for each workspace, like this:

image: node:lts

pipelines:
    default:
        - step:
            name: Build APP
            caches:
              - nodeall
              - nodeapp
              - nodelogin
              - nodeapi
              - yarn
            script:
                - yarn
                - yarn workspace app build
        - step:
            name: Build LOGIN
            caches:
              - nodeall
              - nodeapp
              - nodelogin
              - nodeapi
              - yarn
            script:
                - yarn
                - yarn workspace login build
        - step:
            name: Build API
            caches:
              - nodeall
              - nodeapp
              - nodelogin
              - nodeapi
              - yarn
            script:
                - yarn
                - yarn workspace api build


definitions:
  caches:
    nodeall: ./node_modules
    nodeapp: ./packages/app/node_modules
    nodelogin: ./packages/login/node_modules
    nodeapi: ./packages/api/node_modules
    yarn: /usr/local/share/.cache/yarn

[Alexey Grinko]

This one finally helped in my case, too. Thank you!

[Damon Miller]

I am not using yarn workspaces and still have the same issue.

I have had to setup custom node and yarn caches in order to get my yarn install step to complete in <1s.

definitions:
  caches:
    nodecustom: ./node_modules
    yarncustom: /usr/local/share/.cache/yarn

I have had an open ticket with support around yarn caching, including a link to this thread and my solution, and got them to open a ticket to add native support for it.

https://jira.atlassian.com/browse/BCLOUD-20511

[Artem Loginov]

For yarn berry:

Next string need to be added to scripts block as yarn berry uses local cache by default

- yarn config set enableGlobalCache true

Definitions block

caches:
  nodecustom: ./node_modules
  yarn: /root/.yarn/berry

PS

In case /root/.yarn/berry folder doesn't work  you can look for current cache folder with next command in scripts block

- yarn config get globalFolder

[Attila Szeremi]

As the person with the most liked comment here, please see my updated comment on how to get the fastest yarn installs on average in 2023.

[Drew Gallagher]

For yarn v1, can we just use the

yarn cache dir 

instead of the 

node_modules 

 folder?

Or are they both needed to maximize speed?

[williambell9708]

that is good for sure.

[Drew Gallagher]

I noticed that when I use a shared cache

./node_modules

it does not speed up the build if i have multiple different envs using it. 

i had to revert to having a node modules cache for each env. 

is there a reason I would have to do this?

Also these builds don't use the cache consistently

[williambell9708]

make sure you use `yarn cache dir` to  cache that directory.

[Drew Gallagher]

yarn cache dir gives the directory of the yarn cache. But it doesn't cache a directory

[Drew Gallagher]

Has anyone discovered the underlying reason why the predefined node cache does not work with yarn install consistently?