Is Yarn supported for bitbucket-pipelines caching?

Shane Fast October 5, 2017

I've been using the node cache feature for some time now and have been using yarn to load my load modules with the following pipeline snippet:

pipelines: 
default:
-
step:
caches:
-
node
script:
- yarn install
.
.
.

Observing the performance over several pipelines I haven't noticed any reduction in load time. I'm aware that the cache may not trigger for every pipeline, but I wanted to know if using yarn instead of npm is currently supported or is planned to be supported by pipelines caching.

4 answers

1 accepted

2 votes
Answer accepted
Ana Retamal
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 10, 2017

Hi Shane, can you confirm if you're managing your dependencies correctly as explained at Manage dependencies with Yarn.

Cheers,

Ana 

Emilio Martinez November 2, 2017

I can't reply for Shane, but I am following the same process outlined in the doc you linked to, but am not seeing any benefits either. I believe I used to before but it's been a while since I noticed no benefits. At this time, my `yarn install`s take over half a minute despite the cache.

Incidentally, at the bottom I see `Cache "node": Skipping upload for previously downloaded cache`

Deleted user November 30, 2017

Once install.sh is downloaded and executed, it will check if Yarn is already installed, if it does it'll just continue to the next script. FYI the script has 6.71 KB in size, so there's not too much to download :)

 

Cheers.

Dan Rumney January 11, 2018

I think these answers are missing the point.

This isn't about Bitbucket caching the yarn installation... it's about caching the modules that yarn installs... the dependencies of the project being built.

I routinely see my `yarn install`s taking ~3m... they should take less than a second if my node_modules directory was being cached properly

Like # people like this
Shane Fast January 19, 2018

I did swap to npm install just for pipeline builds for the reduction in build times, but still use yarn install for my dev environment/production. 

Like aglitchman likes this
Deleted user January 21, 2018

After playing around a bit, I've found that the following currently works

pipelines:
  default:
    - step:
        name: Build
        image: node:6.11.4
        caches:
          - yarncustom
        script:
          - curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.3.2
          - export PATH=$HOME/.yarn/bin:$PATH
          - yarn
          - yarn build
definitions:
  caches:
    yarncustom: /usr/local/share/.cache/yarn/v1

I got here by executing "yarn cache dir" on a Bitbucket Pipeline and then just setup a custom cache as per the documentation. I then executed my build subsequent times with "yarn --verbose". I was able to confirm that builds were using the cache by seeing if it needed to download the modules again.

Like # people like this
Mike_Stead March 1, 2018

@[deleted] Thanks that worked for me too and double checked like you did. The cache download time is almost as slow as pulling from yarn but still a little boost.

Another point, the node docker image contains yarn so don't think there's a reason to install it via curl.. unless I'm missing something.

Like Archit T likes this
Alex Bergin July 4, 2018

Yeah, in pipelines, I've found yarn sadly very slow, even with the cache workaround from @[deleted], and have switched back to npm install. Thanks for trying to help all.

sondrel January 16, 2019

I was just about to try this out myself, but yesterday the `yarn install` step started executing in 1 second.

Seams like they might have fixed the caching in the pipelines, can anyone else confirm this?

nakul_predixt February 3, 2019

Using 'yarn' to install all dependencies used to take around 6mins a couple of months ago -> today it only took 40s. After that it took 1s so it seems like the caching is working. 

Oscar Origen February 23, 2019

Hi @nakul_predixt and @sondrel could you share your config file. My yarn command still take like 60-90s to execute.

Oscar Origen February 23, 2019

The normal `-node` cache doesn't work with yarn and triggers the whole installation. 

However a custome cache: ` nodecustom: ./node_modules` works well with yarn and it just takes about 1s. 

Like # people like this
sondrel February 26, 2019

@Oscar OrigenDefault pipeline below:

image: node:11

pipelines:
default:
- step:
caches:
- node
script:
- yarn
- yarn test
- yarn build
Like # people like this
Arthur Simon March 8, 2019

`yarncustom: ./node_modules` solution worked for me.

But the first time you run it will be slow because it will be building the cache.

Subsequent deployments will be quick.

Attila Szeremi December 3, 2019

Here's what I needed to do in December 2019 to get Yarn to take as fast as <1s.

image: node:10.17.0

pipelines:
default:
- step:
name: Packages
caches:
- nodecustom
- yarn
script:
- yarn

definitions:
caches:
nodecustom: ./node_modules
yarn: /usr/local/share/.cache/yarn

Yes, nodecustom is necessary as per @Oscar Origen . Just "node" is not good enough. Don't ask why.

For yarn caching, you don't include the /v1 bit. At this time /v6 is being used; it's simpler if we just cache the version's parent directory to future-proof.

 

2023 October edit:

Now that Bitbucket Pipelines has file-based cache key support, the way to get fastest CI runs on average (1s for the yarn install part most of the time) is to use file-based caching for node_modules:
 

image: node:10.17.0

pipelines:
default:
- step:
name: Packages
caches:
- nodecustom
script:
- yarn

definitions:
caches:
nodecustom:
key:
files:
- yarn.lock
path: ./node_modules

This way even if you're working on different branches where yarn.lock is different, it'll still only install the dependencies for real 1x per yarn.lock hash, and yarn install will take 1s from then on on both branches without the situation where only one branch gets 1s installation times and the other requires a proper install. (up to the cache expiry of 1 week of course).

Also, notice how I'm not using the yarn shared cache anymore. This is intentional. I don't recommend using it at all, because using it adds ~30 seconds your CI time for loading that data for every run unnecessarily, even if yarn install would take 1s either way. Sure, fresh installs would become faster with the yarn shared cache, but 95% of the time you'll just be hitting the nodecustom cache and get 1s yarn install times.

Like # people like this
Justin Hamade February 4, 2020

Thanks @Attila Szeremi This should be the accepted answer.

Justin McMahon March 6, 2020

@Attila Szeremi's solution appears to work unless you're using yarn workspaces. I've spent almost two work days trying to figure out how to cache dependencies when using yarn workspaces and have come up with nothing. I'd appreciate any ideas.

 

Also, for what it's worth, Bitbucket Pipelines takes ~13mins to run all my steps whereas AWS CodeBuild runs the same pipeline with no cache in under 5mins.

Like # people like this
chason May 5, 2020

I have the same troubles as @Justin McMahon i'm using yarn workspace and yarn install is still 1min not 1s.

Like # people like this
viljar.kargenberg August 26, 2020

I managed to get the yarn to run in 1s when using yarn workspaces by creating additional caches for each workspace, like this:

 

image: node:lts

pipelines:
default:
- step:
name: Build APP
caches:
- nodeall
- nodeapp
- nodelogin
- nodeapi
- yarn
script:
- yarn
- yarn workspace app build
- step:
name: Build LOGIN
caches:
- nodeall
- nodeapp
- nodelogin
- nodeapi
- yarn
script:
- yarn
- yarn workspace login build
- step:
name: Build API
caches:
- nodeall
- nodeapp
- nodelogin
- nodeapi
- yarn
script:
- yarn
- yarn workspace api build


definitions:
caches:
nodeall: ./node_modules
nodeapp: ./packages/app/node_modules
nodelogin: ./packages/login/node_modules
nodeapi: ./packages/api/node_modules
yarn: /usr/local/share/.cache/yarn
Like # people like this
Alexey Grinko September 15, 2020

This one finally helped in my case, too. Thank you!

Damon Miller October 8, 2020

I am not using yarn workspaces and still have the same issue.

I have had to setup custom node and yarn caches in order to get my yarn install step to complete in <1s.

definitions:
caches:
nodecustom: ./node_modules
yarncustom: /usr/local/share/.cache/yarn

 

I have had an open ticket with support around yarn caching, including a link to this thread and my solution, and got them to open a ticket to add native support for it.

https://jira.atlassian.com/browse/BCLOUD-20511

Like # people like this
Artem Loginov June 30, 2022

For yarn berry:

 

Next string need to be added to scripts block as yarn berry uses local cache by default

- yarn config set enableGlobalCache true

Definitions block

caches:
nodecustom: ./node_modules
yarn: /root/.yarn/berry

 

PS

In case /root/.yarn/berry folder doesn't work  you can look for current cache folder with next command in scripts block

 

- yarn config get globalFolder
Like soldeplata likes this
Attila Szeremi October 16, 2023

As the person with the most liked comment here, please see my updated comment on how to get the fastest yarn installs on average in 2023.

0 votes
Drew Gallagher February 23, 2023

For yarn v1, can we just use the

yarn cache dir 

instead of the 

node_modules 

 folder?

 

Or are they both needed to maximize speed?

williambell9708 June 5, 2023

that is good for sure.

0 votes
Drew Gallagher February 23, 2023

I noticed that when I use a shared cache

./node_modules

it does not speed up the build if i have multiple different envs using it. 

i had to revert to having a node modules cache for each env. 

is there a reason I would have to do this?

Also these builds don't use the cache consistently

williambell9708 June 5, 2023

make sure you use `yarn cache dir` to  cache that directory.

Drew Gallagher October 5, 2023

yarn cache dir gives the directory of the yarn cache. But it doesn't cache a directory

0 votes
Drew Gallagher February 17, 2023

Has anyone discovered the underlying reason why the predefined node cache does not work with yarn install consistently?

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events