Runners are getting queued in runners-autoscaler for kubernetes.

[Hafeez Ghanchi]

We are currently experiencing an issue with our Bitbucket Pipelines runners utilizing the runner-autoscaler for Kubernetes in GKE. Pipelines are not targeting available runners in parallel, leading to significant delays and queuing of builds, even when resources are available in the cluster. Specifically, we observe the following behavior:

• Only one runner is active and running a pipeline at any given time.
• Subsequent pipelines remain in a queued state, despite the availability of additional runners in the Kubernetes cluster.
• The system appears unable to connect newly scaled runners to the pending pipelines.
  
  

We have already performed the following troubleshooting steps without resolving the issue:

1. Increased the min runner count in the autoscaler configuration.
2. Updated the runner-autoscaler version to the latest available.
3. Adjusted the scale_up_threshold parameter.

This behavior occurs even when the autoscaler successfully scales up new runner pods in the GKE cluster.

We expect the Bitbucket Pipelines to be able to target and utilize multiple available runners in parallel as soon as they are scaled up by the autoscaler, allowing concurrent execution of queued pipelines.