Speeding up pipeline step execution times ?

[Adam Brunt]

Hi all,

So I understand that each step in a pipeline runs on it's own build agent [ hence why you can't easily share files/data/variables between steps ].

Today highlighted an issue, to me, about what seems to be increasingly longer "wait times" for a Bitbucket pipeline run; we had a pipeline run that was showing as started "10 minutes ago" but a duration of "1m 10s" - there had been no failed/restarted steps in this run - which suggests to me that across the 4 steps (it had processed at that point) it had taken 8m 50s to find available agents.

The first question that springs to mind is to confirm that "waiting time" does not count as build minutes ?

The next question is to see if anyone has any ideas on how to reduce the amount of "waiting time".

The obvious solution would be to reduce the number of steps in the pipeline. However, there are two issues with doing that in our scenario; a few of the steps are exported as custom pipelines as they are also needed by pipelines in other repos and - secondly - many of the steps just run pipes and I was under the impression that a single step couldn't mix and match script and pipes.

Other than reducing the number of steps, is self-hosting the only alternative to reducing the "wait time" ?

[Patrik S]

Hello @Adam Brunt ,

thanks for reaching out!

As you correctly stated, when a pipeline starts, the steps will be executed sequentially, with each step being run on its own isolated agent.

When a step is triggered, it will first start looking for an available agent. This state is called "Queued" state and does not count against the build minutes because the step hasn't actually been scheduled to an agent. You can identify the queued state in the UI during a step's execution : 

Once an available agent is found, the step will change it's state to Running. When in that state, the step starts consuming the build minutes. It's important to note that the Build setup phase(where docker image is pulled, repository is cloned inside the container, artifact and caches are downloaded, etc) happens when the step is already in the Running state, so it's considered for the build minutes.

With that in mind, if by "wait time" you were referring to the time where the step is Queued looking for an Agent, that time may vary based on the current load of Atlassian infrastructure and availability of new agents. As you mentioned, one alternative could be reduce the number of steps, so there's less lookup for new agents. If that's an option for your use-case and you were mostly concerned about being able to use pipes and script together, that's indeed possible, as pipes are considered as normal command in the script's section: 

pipelines:
 default:
    - step:
      name: Lint the template
      image: python:3.8
      script:
        - echo "test"
        - pipe: atlassian/aws-cloudformation-deploy:0.12.0
          variables:
            AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
            AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
            AWS_DEFAULT_REGION: 'us-east-1'
            STACK_NAME: 'my-stack-name'
            TEMPLATE: 'template.yaml'
            WAIT: 'true'
        - echo "test"
        - pipe: atlassian/aws-s3-deploy:1.6.1
          variables:
          AWS_ACCESS_KEY_ID: '<string>' # Optional if already defined in the context or OIDC used.
          AWS_SECRET_ACCESS_KEY: '<string>' # Optional if already defined in the context or OIDC used.
          AWS_DEFAULT_REGION: '<string>' # Optional if already defined in the context or OIDC used.
          AWS_OIDC_ROLE_ARN: '<string>' # Optional by default. Required for OpenID Connect (OIDC) authentication.
          AWS_ROLE_ARN: '<string>' # Optional. Required for STS authentication.
          AWS_ROLE_SESSION_NAME: '<string>' # Optional. Required for STS authentication.
          S3_BUCKET: '<string>'
          LOCAL_PATH: '<string>'

Another alternative would be to use self-hosted runners, where you could have more control over the number of agents you have available.

That being said, Pipelines infrastructure scales according to the demand, so you shouldn't be waiting much time for a new agent in the Queued state. If you found this is occurring more frequently and started recently, I would suggest opening a support ticket with us so we can take a deeper look at the affected builds.

Thank you, @Adam Brunt !

Patrik S

[Adam Brunt]

Hi @Patrik S

Thanks for the clarification about the build minutes and clearing up that you can run script and pipes in the same step [ this would at least make our YAML file a lot cleaner as well :) ]

One problem we have running self hosted steps would be the tests we run in a particular step. These are MXUnit based tests which generate XML files that, currently, the Bitbucket agent is able to interpret and report on which tests have failed (in the "Tests" tab of the pipeline console window). If we were to move this step as is to a self-hosted runner, would the pipeline console still show which tests had failed ?

Also, would the whole pipeline have to run self-hosted ? Or can you run individual steps self-hosted whilst running others on a Bitbucket agent ?

[Patrik S]

Hello @Adam Brunt ,

When using self-hosted runners, there will be no changes to how you see logs in the UI. The runners stream the logs from your local run to Bitbucket Pipelines, and you'll see them in the same way you see for builds running in Atlassian runners. This includes build logs, artifacts, test results, etc.

As for your second question, the YAML configuration that defines if the execution will happen at atlassian runners of self-hosted runners is done at the step-level, so it's totally possible to have a set of steps running on self-hosted, while other still runs in atlassian runners.

E.g.:

- step:
      name: Step 1
      runs-on: 
        - 'self.hosted'
        - 'my.custom.label'
      script:
        - echo "This step will run on a self hosted runner.";
- step:
      name: Step 2
      script:
        - echo "This step will run on Atlassian's infrastructure as usual.";

You can find more details on how to configure runners in your YAML at the following documentation:

• Configure your runner in bitbucket-pipelines.yml 

Thank you, @Adam Brunt !

Patrik S