Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,300,116
Community Members
 
Community Events
165
Community Groups

Need solution for these runner limitation

Hi Team,

==> What is Atlassian's plan to resolve below issue? Is there any timeline for this?

  • Each machine can have multiple runners running with different names.
  • But out of these multiple runners only one can be used for pipeline as each runner will try to clone the repository in /tmp/build folder.
  • Ideally in /tmp/build/ folder individual folder should be created for each pipeline run (maybe timestamped based).

==> What is the plan for broadening the scope of the points mentioned below?

  • Each runners are tied to pipeline at repository level.
  • There is no allocation of runners at workspace level.
  • If there are 20 teams who need runners at a time, we need to have 20 machine for each of them.
  • Since each runner is tied to a repository, most of the time it is going to be idle as it can not be shared by any other team/project/repository.

==> Please provide your inputs for below query. This is a big limitation as an enterprise we expect ~5-8 developers triggering job simultaneously on a single repo. and just FYI we have more than 100 repos and many will be using runner for processing. 

  • Suppose two developers trigger a build on the same branch or different branch simultaneously, what will happen in this case as the source code is cloned in /tmp/build.  As of now it doesn't allow us to trigger new job and pauses itself.

==> Can we run multiple runners over a kubernetes cluster? If yes, can you please redirect me to the documentation.

 

Thanks, 

Akshay

1 answer

0 votes

@Akshay Sharma Sorry for the late reply, it slipped off my radar.

==> What is Atlassian's plan to resolve below issue? Is there any timeline for this?

  • Each machine can have multiple runners running with different names.
  • But out of these multiple runners only one can be used for pipeline as each runner will try to clone the repository in /tmp/build folder.
  • Ideally in /tmp/build/ folder individual folder should be created for each pipeline run (maybe timestamped based).

The ability to execute multiple runners on the same machine is something that we would like to add but it will be after the EAP. Since we are still prioritizing the features I can't give you a timeline on when it will be added. Would the ability to execute the runner on a Kubernetes cluster solve this problem?

==> What is the plan for broadening the scope of the points mentioned below?

  • Each runners are tied to pipeline at repository level.
  • There is no allocation of runners at workspace level.
  • If there are 20 teams who need runners at a time, we need to have 20 machine for each of them.
  • Since each runner is tied to a repository, most of the time it is going to be idle as it can not be shared by any other team/project/repository.

We are planning to add workspace runners soon after the EAP. I will be able to give you a timeline as we get closer to GA.

==> Please provide your inputs for below query. This is a big limitation as an enterprise we expect ~5-8 developers triggering job simultaneously on a single repo. and just FYI we have more than 100 repos and many will be using runner for processing. 

  • Suppose two developers trigger a build on the same branch or different branch simultaneously, what will happen in this case as the source code is cloned in /tmp/build.  As of now it doesn't allow us to trigger new job and pauses itself.

With workspace runners you will be able to reuse runners across repositories, I hope that will solve the problem of ~5-8 developers triggering the job at the same time. If a runner is not available the pipeline job will wait till it's available. `/tmp/build` is the default working directory and it can be changed. Again, would the ability to execute the runner on a Kubernetes cluster solve this problem?

==> Can we run multiple runners over a kubernetes cluster? If yes, can you please redirect me to the documentation.

We have not yet tested running a runner in a Kubernetes cluster, in theory, it should be possible. I will keep you posted when we have documentation around it.

Please let me know if you have any further questions.

Hi @Justin Thomas  thanks a lot for your reply.

Yes. it would help us in great capacity if we can get these at the earliest.

Also kubernetes solution will help and unblock us greatly. Can you please share whatever rough documentation or approach available with you for running the runnner in k8s cluster. I would like to start the PoC on that.

@Akshay Sharma You can start the runner on a Kubernetes cluster with the following template

apiVersion: v1
kind: List
items:
- apiVersion: v1
kind: Secret
metadata:
name: runner-oauth-credentials
# labels:
# accountUuid: # Add your account uuid to optionally allow finding the secret for an account
# repositoryUuid: # Add your repository uuid to optionally allow finding the secret for a repository
# runnerUuid: # Add your runner uuid to optionally allow finding the secret for a particular runner
data:
oauthClientId: # add your base64 encoded oauth client id here
oauthClientSecret: # add your base64 encoded oauth client secret here
- apiVersion: batch/v1
kind: Job
metadata:
name: runner
spec:
template:
# metadata:
# labels:
# accountUuid: # Add your account uuid to optionally allow finding the pods for an account
# repositoryUuid: # Add your repository uuid to optionally allow finding the pods for a repository
# runnerUuid: # Add your runner uuid to optionally allow finding the pods for a particular runner
spec:
containers:
- name: runner
image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
env:
- name: ACCOUNT_UUID
value: # Add your account uuid here
- name: REPOSITORY_UUID
value: # Add your repository uuid here
- name: RUNNER_UUID
value: # Add your runner uuid here
- name: OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
name: runner-oauth-credentials
key: oauthClientId
- name: OAUTH_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: runner-oauth-credentials
key: oauthClientSecret
- name: WORKING_DIRECTORY
value: "/tmp"
volumeMounts:
- name: tmp
mountPath: /tmp
- name: docker-containers
mountPath: /var/lib/docker/containers
readOnly: true # the runner only needs to read these files never write to them
- name: var-run
mountPath: /var/run
- name: docker-in-docker
image: docker:20.10.5-dind
securityContext:
privileged: true # required to allow docker in docker to run and assumes the namespace your applying this to has a pod security policy that allows privilege escalation
volumeMounts:
- name: tmp
mountPath: /tmp
- name: docker-containers
mountPath: /var/lib/docker/containers
- name: var-run
mountPath: /var/run
restartPolicy: OnFailure # this allows the runner to restart locally if it was to crash
volumes:
- name: tmp # required to share a working directory between docker in docker and the runner
- name: docker-containers # required to share the containers directory between docker in docker and the runner
- name: var-run # required to share the docker socket between docker in docker and the runner
# backoffLimit: 6 # this is the default and means it will retry upto 6 times if it crashes before it considers itself a failure with an exponential backoff between
# completions: 1 # this is the default the job should ideally never complete as the runner never shuts down successfully
# parallelism: 1 # this is the default their should only be one instance of this particular runner

You can only run one runner per job. It is still pretty early days for the runners, and I won't be surprised if the above template changes. Please let me know if the template works for you.

Suggest an answer

Log in or Sign up to answer
TAGS

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you