Pipelines Build Setup taking too long

[Paul Cable]

I've been playing around with pipelines and have managed to get my pipeline working exactly the way I want. However it takes 5 minutes to run and looking back through the logs most of this is due to the Build Setup portion of each step.

My pipeline has 3 steps and the Build Setup takes between 30s and 90s on each step. 

I was using a private docker image but have changed this to be a public image as I understand these are cached however this hasn't affected performance.  The image is large (650MB). The repo is about 25mb.

Any ideas of what I can change to improve / diagnose the performance issues?

image:
  name: pjcable/temp:meteor-base-v3
  #username: pjcable
  #email: paul@highspecsolutions.com
  #password: $DOCKER_HUB_PASSWORD

options:
  max-time: 5

pipelines:
  branches:
    master:

      - step:
          name: Testing
          max-time: 3
          caches:
            - composer
          script:
            - cp .env.example .env
            - composer install
            - php artisan key:generate
            - sleep 10
            - vendor/bin/phpunit
            - php artisan serve &
            - php artisan dusk
            - kill $(jobs -p)
          services:
            - mysql
            - dusk-runner

      - step:
          name: Generate production assets
          caches:
            - node
          script:
            - echo "<?php return [ 'number' => ${BITBUCKET_BUILD_NUMBER}];" > config/version.php
            - npm install
            - npm run production
            - git config --global user.name "Meteor Bot"
            - git config --global user.email "paul+meteor-bot@highspecsolutions.com"
            - git commit -am "Committing V${BITBUCKET_BUILD_NUMBER} production assets"
            - git tag "V${BITBUCKET_BUILD_NUMBER}"
            - git remote remove origin
            - git remote add origin https://${BB_AUTH_STRING}@bitbucket.org/meteorsg/web.git
            - git push origin "V${BITBUCKET_BUILD_NUMBER}"
      - step:
          name: Deploy to staging
          deployment: staging
          #trigger: manual
          script:
            - echo "Push to staging server"
            - ssh -v bitbucket@meteor.highspecsolutions.com "echo hello"
            - envoy run deploy-staging --verbose --release_tag=V${BITBUCKET_BUILD_NUMBER}

definitions:
  services:
    mysql:
      image: mysql:5.7
      environment:
        MYSQL_DATABASE: 'homestead'
        MYSQL_USER: 'homestead'
        MYSQL_PASSWORD: 'secret'
        MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
    dusk-runner:
       image: pjcable/dusk-runner:latest

[Jens De Pelsmaeker]

Hello, how can this be neglected for so long? This is a crucial bottleneck in effectively using Bitbucket Pipelines. We are losing minutes every day and therefore hours every month because of this. Please give this ticket the attention it deserves?

[Deleted user]

We have the same issue. Pipeline's build setup speed varies from 30 seconds to 50 minutes!

This is the output of the build setup and unfortunately there is no timing being printed out in details except the time took for downloading caches. However, the whole step took 46 minutes:

+ umask 000+ GIT_LFS_SKIP_SMUDGE=1 git clone --branch="CT-123-change-parameter-name-in-entity" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/abcd/abcd.git $BUILD_DIR ; git reset --hard 8bdfe2cf4dd33902b321208d92a90f8f236d3660 ; git remote set-url origin git@bitbucket.org:abcd/abcd.git
Cloning into '/opt/atlassian/pipelines/agent/build'...
HEAD is now at 8bdfe2cf4 CT-123: issue description 
+ chmod 777 $BUILD_DIR
Cache "maven": Downloading
Cache "maven": Downloaded 226 MB in 14 seconds
Cache "maven": Extracting
Cache "maven": Extracted in 3 seconds
Cache "nodemodules": Downloading
Cache "nodemodules": Downloaded 4 MB in 1 seconds
Cache "nodemodules": Extracting
Cache "nodemodules": Extracted in 0 seconds

this not only costs us but also delays the development team. Any idea?

 

[Fei Zhang]

Any update on this question?  

I'm raising a task. https://jira.atlassian.com/browse/BCLOUD-19451

[Александр Сосновский]

Still exists

[Matthew Balshaw]

It's extremely frustrating that we're being BILLED per minute that we have to wait on bitbucket to setup its own infrastructure.

Sometimes a step for me takes 15 minutes to "Setup" but only 1 minute to run. It feels extremely unjust to charge us for Atlassians own serverless issues. It can't be something we've done, as often that setup step will be instant.

We're using the Atlassian firebase deploy pipe for reference.

Our company already moved away from JIRA to another project management platform because it was just consistently extremely slow. It's looking more like we'll need to migrate off bitbucket too, given how slow merging branches and running pipelines has become in the last months. (Across more than 3 medium sized projects on different tech stacks).

It'd be nice if performance was a bigger priority at Atlassian.

[Jonathan H_ Wage]

I have the same issue. We are a paid customer and sometimes the Build setup step at the very beginning takes 2+ minutes.

[Chris Paynter]

Yes, this issue is starting to become fairly frustrating for me also.

[Alexander Roddis]

Realy Attlassian? Realy...no reply, no help...nice support!

[JasonGilholme]

I agree - my most recent pipeline was 4min & 41 seconds of which <50 seconds were tests that I want to run.

If i'm going to be continually billed for ~5x the amount of machine time I'm using, I'll just solve this problem with a githook and a bash script.

[JasonGilholme]

The variable time for build setup is ridiculous - sometimes it's 10 seconds, other times it's 10 minutes.

If I hadn't have already clocked over 50 build minutes and been charged overages I'd already be turning it off.

[Ryan Cole]

I know that the underlying architecture of pipelines is serverless, which makes me wonder if that's not the source of the slowness and variability...

[Yash Dave]

I was able to save few seconds by setting artifacts download to false to all steps where artifacts are not needed

- step: &set-build-env-step

    name: 'Set build env'

    caches:

        - yarn

    artifacts:

        download: false

         paths:

             - '.env'

[Nikolay Kostov]

Do you know if the selfhosted runners are faster?

I would actually prefer to pay for a small VM to do all my jobs!

[Shaveta C]

I am trying on SelfHosted server, it is strange that sometime it take few minutes to fetch repo.  But sometimes it take hours. 
Not sure what is the fix here.

[Anton Samokhvalov]

I noticed that every Build setup stage of every step in pipeline takes a lot of time. So, when I combine all my 3 steps into one with flat script inside, total time of deployment reduced 3 times too.
But still, "Build setup" stage takes half of time of the whole process.

[Pedro Rodrigues]

3 years and not even a single reply...?

[Anthony Janicas]

Same problem here... this definitely needs to be addressed.

[Bo Zheng]

Thank Atlassian, I am sitting right now in front of the laptop waiting for the pipeline and now the berlin time is nearly 20:00 in the evening. U can do nothing but wasting time 🤢

[Thomas Hui]

using default image 2, same issue

[Indie Hortelano]

Any update?

[Александр Сосновский]

Any news?

[Amir Harel]

We encounter same issue. Performance is really unstable - same pipeline might take 5 minutes in one run or 1 hour in another. Making me think if this was the right tool to use. 

[Shaveta C]

This is happening for me on SelfHosted Pipeline not on MicrosoftHosted pipeline

[Petr Peller]

We have the same issue. Steps are nice thing and it's probably useful in some cases that each step is a new docker image, but in our (and perhaps most?) use cases there is really no benefit and waiting 30s each step is not optimal.

[Danyal Iqbal]

Conventional wisdom would provide resources to the environment. However, docker builds are conventionally slow. (Docker is not the rockstar, everyone thinks it is:)

You can optimize the build time using the --cache-from feature, see an example:

pipelines:
 default:
 - step:
 services:
 - docker
 script:
 - export IMAGE_NAME=your-docker-hub/your-repo:$BITBUCKET_BRANCH
 - docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
 - docker pull $IMAGE_NAME || true
 - docker build --cache-from $IMAGE_NAME -t $IMAGE_NAME .
 - docker push $IMAGE_NAME

Avoid using the pipeline docker cache, only wastes time... :)

check https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html for more infos.

[Paul Cable]

Thanks Danyal however this doesn't address the issue i'm looking at.

The setup you show and the docker cache only apply to when you're building docker images within your pipeline.

The image that takes forever to load for me is actually the custom build agent image.