How to ensure all the steps of a pipeline is executed on the same runner

Question

On my self hosted runner, I am running a pipeline that consists of several steps. I noticed that the steps are randomly executed on different runners running on different servers.

But I want to run the entire pipeline on the same self hosted runners, otherwise it will fail due to missing dependencies from the previous steps.

How can I ensure that once a pipeline starts, all the steps are executed on the same runner?

Mark C · Answer

Hi @DARSHAN GAD

I don't think it is possible at the moment.

Perhaps you can use Docker Linux Runners in this case, however, you'll need to define your preferred environment in each build step.

Regards,
Mark C

DARSHAN GAD · Answer

@Mark C Yes we could use single runner, but that would defeat the purpose. We have several servers running sever runners.

Having several runners enable faster running of pipelines since many developers are working in parallel in branches.

Two of the steps in our pipeline looks like below.

* - step: *checkout_code

* - step: *build_software

the problem is that currently these two steps can be executed in random runners.  But that would not make sense because the result can be that the code is checked out in one runner running a server and then build software in another runner in another server, giving unpredictable results.

To solve this, I am forced to merge the two steps into one step so that it checks out the code and builds on the same runner/server. The consequence is that, our huge code base takes about 20 minutes to checkout(which usually succeeds) and then errors can occur in the build step. This means that I have to wait for more than 20 minutes each time to get the result. Sometimes I just want to re-run the failed step and go to the build step directly which I am not able to do because of the problem described above.

I clearly see a need to enforce a step in the runner of my choice (random / same as all other steps of a pipeline). Is this possible or could it be implemented as an option in the future?

Regards,

Darshan

Paul Hicks · Answer

I'm keen to know the answer to the original question too. I would not be able to use a single runner: I want to be able to run multiple pipelines at once, and to have all steps with matching runs-on properties to run on the same runner. Being able to set `clone: enable: false` on steps would greatly speed things up.

To make things a bit more complicated, my pipelines have Linux self-hosted, Windows self-hosted, and normal in-cloud steps. I would like to be able to rely on the file system for steps running on self-hosted runners persisting for the duration of the pipeline.