How do the caches work for Pipeline Runners?

ntmat August 30, 2021

Hi,

I'm running a group of tests with the Runner feature of Bitbucket Pipelines. Inside the runners I have steps to install pip requirements for testing and have specified pip as a cache for the runners steps. However the runners don't seem to be pulling these requirements from cache on subsequent runs.

Can anyone point out where I'm going wrong in the high level configuration below:

pull-requests:
"**":
- parallel:
- step:
name: solution_testing_0
runs-on:
- self.hosted
- linux
- solution.testing
caches:
- pip
script:
- bash install_system_requirements.sh # Installs all pip requirements
- bash run_test.sh --group 0 # Runs subset of tests
- step:
name: solution_testing_1
runs-on:
- self.hosted
- linux
- solution.testing
caches:
- pip
script:
- bash install_system_requirements.sh # Installs all pip requirements
- bash run_test.sh --group 1 # Runs subset of tests


How are caches supposed to be configured for pipeline runners?

1 answer

1 accepted

0 votes
Answer accepted
Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 31, 2021

@ntmat Caches work the same way for both cloud and self-hosted runners. Be aware that the performance improvement from caches is different between cloud and self-hosted runners. Self-hosted runners are located in your data center, so it might take more time to pull the cache from Bitbucket. 

ntmat August 31, 2021

@Justin ThomasThanks for your reply. I can confirm that the cloud and self-hosted runners use the cache as expected, I needed to delete the existing cache so the runner would re-upload and use the new pip cache.

I see your point regarding performance, what is the alternative? Would you recommend using a docker:dind service to startup a local docker image with all the necessary dependencies preinstalled to avoid cache download from Bitbucket for self.hosted runners?

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 31, 2021

Unfortunately, there is no easy way around it. Using a docker:dind service is a workaround and can help, but I think in long term it might complicate the setup. It depends on the performance impact.

Cheers,

Justin

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events