Need solution for these runner limitation

[Akshay Sharma]

Hi Team,

==> What is Atlassian's plan to resolve below issue? Is there any timeline for this?

• Each machine can have multiple runners running with different names.
• But out of these multiple runners only one can be used for pipeline as each runner will try to clone the repository in /tmp/build folder.
• Ideally in /tmp/build/ folder individual folder should be created for each pipeline run (maybe timestamped based).

==> What is the plan for broadening the scope of the points mentioned below?

• Each runners are tied to pipeline at repository level.
• There is no allocation of runners at workspace level.
• If there are 20 teams who need runners at a time, we need to have 20 machine for each of them.
• Since each runner is tied to a repository, most of the time it is going to be idle as it can not be shared by any other team/project/repository.

==> Please provide your inputs for below query. This is a big limitation as an enterprise we expect ~5-8 developers triggering job simultaneously on a single repo. and just FYI we have more than 100 repos and many will be using runner for processing. 

• Suppose two developers trigger a build on the same branch or different branch simultaneously, what will happen in this case as the source code is cloned in /tmp/build.  As of now it doesn't allow us to trigger new job and pauses itself.

==> Can we run multiple runners over a kubernetes cluster? If yes, can you please redirect me to the documentation.

Thanks, 

Akshay

[Justin Thomas]

@Akshay Sharma Sorry for the late reply, it slipped off my radar.

==> What is Atlassian's plan to resolve below issue? Is there any timeline for this?

• Each machine can have multiple runners running with different names.
• But out of these multiple runners only one can be used for pipeline as each runner will try to clone the repository in /tmp/build folder.
• Ideally in /tmp/build/ folder individual folder should be created for each pipeline run (maybe timestamped based).

The ability to execute multiple runners on the same machine is something that we would like to add but it will be after the EAP. Since we are still prioritizing the features I can't give you a timeline on when it will be added. Would the ability to execute the runner on a Kubernetes cluster solve this problem?

==> What is the plan for broadening the scope of the points mentioned below?

• Each runners are tied to pipeline at repository level.
• There is no allocation of runners at workspace level.
• If there are 20 teams who need runners at a time, we need to have 20 machine for each of them.
• Since each runner is tied to a repository, most of the time it is going to be idle as it can not be shared by any other team/project/repository.

We are planning to add workspace runners soon after the EAP. I will be able to give you a timeline as we get closer to GA.

==> Please provide your inputs for below query. This is a big limitation as an enterprise we expect ~5-8 developers triggering job simultaneously on a single repo. and just FYI we have more than 100 repos and many will be using runner for processing. 

• Suppose two developers trigger a build on the same branch or different branch simultaneously, what will happen in this case as the source code is cloned in /tmp/build.  As of now it doesn't allow us to trigger new job and pauses itself.

With workspace runners you will be able to reuse runners across repositories, I hope that will solve the problem of ~5-8 developers triggering the job at the same time. If a runner is not available the pipeline job will wait till it's available. `/tmp/build` is the default working directory and it can be changed. Again, would the ability to execute the runner on a Kubernetes cluster solve this problem?

==> Can we run multiple runners over a kubernetes cluster? If yes, can you please redirect me to the documentation.

We have not yet tested running a runner in a Kubernetes cluster, in theory, it should be possible. I will keep you posted when we have documentation around it.

Please let me know if you have any further questions.

[Akshay Sharma]

Hi @Justin Thomas  thanks a lot for your reply.

Yes. it would help us in great capacity if we can get these at the earliest.

Also kubernetes solution will help and unblock us greatly. Can you please share whatever rough documentation or approach available with you for running the runnner in k8s cluster. I would like to start the PoC on that.

[Justin Thomas]

@Akshay Sharma You can start the runner on a Kubernetes cluster with the following template

apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Secret
    metadata:
      name: runner-oauth-credentials
#      labels:
#        accountUuid: # Add your account uuid to optionally allow finding the secret for an account
#        repositoryUuid: # Add your repository uuid to optionally allow finding the secret for a repository
#        runnerUuid: # Add your runner uuid to optionally allow finding the secret for a particular runner
    data:
      oauthClientId: # add your base64 encoded oauth client id here
      oauthClientSecret: # add your base64 encoded oauth client secret here
  - apiVersion: batch/v1
    kind: Job
    metadata:
      name: runner
    spec:
      template:
#        metadata:
#          labels:
#            accountUuid: # Add your account uuid to optionally allow finding the pods for an account
#            repositoryUuid: # Add your repository uuid to optionally allow finding the pods for a repository
#            runnerUuid: # Add your runner uuid to optionally allow finding the pods for a particular runner
        spec:
          containers:
            - name: runner
              image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
              env:
                - name: ACCOUNT_UUID
                  value: # Add your account uuid here
                - name: REPOSITORY_UUID
                  value: # Add your repository uuid here
                - name: RUNNER_UUID
                  value: # Add your runner uuid here
                - name: OAUTH_CLIENT_ID
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientId
                - name: OAUTH_CLIENT_SECRET
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientSecret
                - name: WORKING_DIRECTORY
                  value: "/tmp"
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                  readOnly: true # the runner only needs to read these files never write to them
                - name: var-run
                  mountPath: /var/run
            - name: docker-in-docker
              image: docker:20.10.5-dind
              securityContext:
                privileged: true # required to allow docker in docker to run and assumes the namespace your applying this to has a pod security policy that allows privilege escalation
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                - name: var-run
                  mountPath: /var/run
          restartPolicy: OnFailure # this allows the runner to restart locally if it was to crash
          volumes:
            - name: tmp # required to share a working directory between docker in docker and the runner
            - name: docker-containers # required to share the containers directory between docker in docker and the runner
            - name: var-run # required to share the docker socket between docker in docker and the runner
        # backoffLimit: 6 # this is the default and means it will retry upto 6 times if it crashes before it considers itself a failure with an exponential backoff between
        # completions: 1 # this is the default the job should ideally never complete as the runner never shuts down successfully
        # parallelism: 1 # this is the default their should only be one instance of this particular runner

You can only run one runner per job. It is still pretty early days for the runners, and I won't be surprised if the above template changes. Please let me know if the template works for you.