Is Yarn supported for bitbucket-pipelines caching?

[Shane Fast]

I've been using the node cache feature for some time now and have been using yarn to load my load modules with the following pipeline snippet:

pipelines: 
  default: 
    - step: 
      caches: 
        - node 
      script: 
        - yarn install
.
.
.

Observing the performance over several pipelines I haven't noticed any reduction in load time. I'm aware that the cache may not trigger for every pipeline, but I wanted to know if using yarn instead of npm is currently supported or is planned to be supported by pipelines caching.

[Ana Retamal · Accepted solution]

Hi Shane, can you confirm if you're managing your dependencies correctly as explained at Manage dependencies with Yarn.

Cheers,

Ana 

[Emilio Martinez]

I can't reply for Shane, but I am following the same process outlined in the doc you linked to, but am not seeing any benefits either. I believe I used to before but it's been a while since I noticed no benefits. At this time, my `yarn install`s take over half a minute despite the cache.

Incidentally, at the bottom I see `Cache "node": Skipping upload for previously downloaded cache`

[Deleted user]

Once install.sh is downloaded and executed, it will check if Yarn is already installed, if it does it'll just continue to the next script. FYI the script has 6.71 KB in size, so there's not too much to download :)

Cheers.

[Dan Rumney]

I think these answers are missing the point.

This isn't about Bitbucket caching the yarn installation... it's about caching the modules that yarn installs... the dependencies of the project being built.

I routinely see my `yarn install`s taking ~3m... they should take less than a second if my node_modules directory was being cached properly

[Shane Fast]

I did swap to npm install just for pipeline builds for the reduction in build times, but still use yarn install for my dev environment/production. 

[Deleted user]

After playing around a bit, I've found that the following currently works

pipelines:
  default:
    - step:
        name: Build
        image: node:6.11.4
        caches:
          - yarncustom
        script:
          - curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.3.2
          - export PATH=$HOME/.yarn/bin:$PATH
          - yarn
          - yarn build
definitions:
  caches:
    yarncustom: /usr/local/share/.cache/yarn/v1

I got here by executing "yarn cache dir" on a Bitbucket Pipeline and then just setup a custom cache as per the documentation. I then executed my build subsequent times with "yarn --verbose". I was able to confirm that builds were using the cache by seeing if it needed to download the modules again.

[Mike_Stead]

@[deleted] Thanks that worked for me too and double checked like you did. The cache download time is almost as slow as pulling from yarn but still a little boost.

Another point, the node docker image contains yarn so don't think there's a reason to install it via curl.. unless I'm missing something.

[Alex Bergin]

Yeah, in pipelines, I've found yarn sadly very slow, even with the cache workaround from @[deleted], and have switched back to npm install. Thanks for trying to help all.

[sondrel]

I was just about to try this out myself, but yesterday the `yarn install` step started executing in 1 second.

Seams like they might have fixed the caching in the pipelines, can anyone else confirm this?

[nakul_predixt]

Using 'yarn' to install all dependencies used to take around 6mins a couple of months ago -> today it only took 40s. After that it took 1s so it seems like the caching is working. 

[Oscar Origen]

Hi @nakul_predixt and @sondrel could you share your config file. My yarn command still take like 60-90s to execute.

[Oscar Origen]

The normal `-node` cache doesn't work with yarn and triggers the whole installation. 

However a custome cache: ` nodecustom: ./node_modules` works well with yarn and it just takes about 1s. 

[sondrel]

@Oscar OrigenDefault pipeline below:

image: node:11

pipelines:
  default:
    - step:
      caches:
        - node
      script:
        - yarn
        - yarn test
        - yarn build

[Arthur Simon]

`yarncustom: ./node_modules` solution worked for me.

But the first time you run it will be slow because it will be building the cache.

Subsequent deployments will be quick.

[Attila Szeremi]

Here's what I needed to do in December 2019 to get Yarn to take as fast as <1s.

image: node:10.17.0

pipelines:
  default:
    - step:
        name: Packages
        caches:
          - nodecustom
          - yarn
        script:
          - yarn

definitions:
  caches:
    nodecustom: ./node_modules
    yarn: /usr/local/share/.cache/yarn

Yes, nodecustom is necessary as per @Oscar Origen . Just "node" is not good enough. Don't ask why.

For yarn caching, you don't include the /v1 bit. At this time /v6 is being used; it's simpler if we just cache the version's parent directory to future-proof.

[Justin Hamade]

Thanks @Attila Szeremi This should be the accepted answer.

[Justin McMahon]

@Attila Szeremi's solution appears to work unless you're using yarn workspaces. I've spent almost two work days trying to figure out how to cache dependencies when using yarn workspaces and have come up with nothing. I'd appreciate any ideas.

Also, for what it's worth, Bitbucket Pipelines takes ~13mins to run all my steps whereas AWS CodeBuild runs the same pipeline with no cache in under 5mins.

[chason]

I have the same troubles as @Justin McMahon i'm using yarn workspace and yarn install is still 1min not 1s.

[viljar.kargenberg]

I managed to get the yarn to run in 1s when using yarn workspaces by creating additional caches for each workspace, like this:

image: node:lts

pipelines:
    default:
        - step:
            name: Build APP
            caches:
              - nodeall
              - nodeapp
              - nodelogin
              - nodeapi
              - yarn
            script:
                - yarn
                - yarn workspace app build
        - step:
            name: Build LOGIN
            caches:
              - nodeall
              - nodeapp
              - nodelogin
              - nodeapi
              - yarn
            script:
                - yarn
                - yarn workspace login build
        - step:
            name: Build API
            caches:
              - nodeall
              - nodeapp
              - nodelogin
              - nodeapi
              - yarn
            script:
                - yarn
                - yarn workspace api build


definitions:
  caches:
    nodeall: ./node_modules
    nodeapp: ./packages/app/node_modules
    nodelogin: ./packages/login/node_modules
    nodeapi: ./packages/api/node_modules
    yarn: /usr/local/share/.cache/yarn

[Alexey Grinko]

This one finally helped in my case, too. Thank you!

[Damon Miller]

I am not using yarn workspaces and still have the same issue.

I have had to setup custom node and yarn caches in order to get my yarn install step to complete in <1s.

definitions:
  caches:
    nodecustom: ./node_modules
    yarncustom: /usr/local/share/.cache/yarn

I have had an open ticket with support around yarn caching, including a link to this thread and my solution, and got them to open a ticket to add native support for it.

https://jira.atlassian.com/browse/BCLOUD-20511

[Artem Loginov]

For yarn berry:

Next string need to be added to scripts block as yarn berry uses local cache by default

- yarn config set enableGlobalCache true

Definitions block

caches:
  nodecustom: ./node_modules
  yarn: /root/.yarn/berry

PS

In case /root/.yarn/berry folder doesn't work  you can look for current cache folder with next command in scripts block

- yarn config get globalFolder

[Drew Gallagher]

For yarn v1, can we just use the

yarn cache dir 

instead of the 

node_modules 

 folder?

Or are they both needed to maximize speed?

[Drew Gallagher]

I noticed that when I use a shared cache

./node_modules

it does not speed up the build if i have multiple different envs using it. 

i had to revert to having a node modules cache for each env. 

is there a reason I would have to do this?

Also these builds don't use the cache consistently

[Drew Gallagher]

Has anyone discovered the underlying reason why the predefined node cache does not work with yarn install consistently?