Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Pipelines Build Setup taking too long

Edited

I've been playing around with pipelines and have managed to get my pipeline working exactly the way I want. However it takes 5 minutes to run and looking back through the logs most of this is due to the Build Setup portion of each step.

My pipeline has 3 steps and the Build Setup takes between 30s and 90s on each step. 

I was using a private docker image but have changed this to be a public image as I understand these are cached however this hasn't affected performance.  The image is large (650MB). The repo is about 25mb.

Any ideas of what I can change to improve / diagnose the performance issues?

 

image.png

image:
name: pjcable/temp:meteor-base-v3
#username: pjcable
#email: paul@highspecsolutions.com
#password: $DOCKER_HUB_PASSWORD

options:
max-time: 5

pipelines:
branches:
master:

- step:
name: Testing
max-time: 3
caches:
- composer
script:
- cp .env.example .env
- composer install
- php artisan key:generate
- sleep 10
- vendor/bin/phpunit
- php artisan serve &
- php artisan dusk
- kill $(jobs -p)
services:
- mysql
- dusk-runner

- step:
name: Generate production assets
caches:
- node
script:
- echo "<?php return [ 'number' => ${BITBUCKET_BUILD_NUMBER}];" > config/version.php
- npm install
- npm run production
- git config --global user.name "Meteor Bot"
- git config --global user.email "paul+meteor-bot@highspecsolutions.com"
- git commit -am "Committing V${BITBUCKET_BUILD_NUMBER} production assets"
- git tag "V${BITBUCKET_BUILD_NUMBER}"
- git remote remove origin
- git remote add origin https://${BB_AUTH_STRING}@bitbucket.org/meteorsg/web.git
- git push origin "V${BITBUCKET_BUILD_NUMBER}"
- step:
name: Deploy to staging
deployment: staging
#trigger: manual
script:
- echo "Push to staging server"
- ssh -v bitbucket@meteor.highspecsolutions.com "echo hello"
- envoy run deploy-staging --verbose --release_tag=V${BITBUCKET_BUILD_NUMBER}

definitions:
services:
mysql:
image: mysql:5.7
environment:
MYSQL_DATABASE: 'homestead'
MYSQL_USER: 'homestead'
MYSQL_PASSWORD: 'secret'
MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
dusk-runner:
image: pjcable/dusk-runner:latest

 

 

17 answers

6 votes
Deleted user Aug 01, 2018

We have the same issue. Pipeline's build setup speed varies from 30 seconds to 50 minutes!

This is the output of the build setup and unfortunately there is no timing being printed out in details except the time took for downloading caches. However, the whole step took 46 minutes:

 

+ umask 000+ GIT_LFS_SKIP_SMUDGE=1 git clone --branch="CT-123-change-parameter-name-in-entity" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/abcd/abcd.git $BUILD_DIR ; git reset --hard 8bdfe2cf4dd33902b321208d92a90f8f236d3660 ; git remote set-url origin git@bitbucket.org:abcd/abcd.git
Cloning into '/opt/atlassian/pipelines/agent/build'...
HEAD is now at 8bdfe2cf4 CT-123: issue description
+ chmod 777 $BUILD_DIR
Cache "maven": Downloading
Cache "maven": Downloaded 226 MB in 14 seconds
Cache "maven": Extracting
Cache "maven": Extracted in 3 seconds
Cache "nodemodules": Downloading
Cache "nodemodules": Downloaded 4 MB in 1 seconds
Cache "nodemodules": Extracting
Cache "nodemodules": Extracted in 0 seconds

this not only costs us but also delays the development team. Any idea?

 

 image.png

Any update on this question?  

I'm raising a task. https://jira.atlassian.com/browse/BCLOUD-19451

I have the same issue. We are a paid customer and sometimes the Build setup step at the very beginning takes 2+ minutes.

Yes, this issue is starting to become fairly frustrating for me also.

It's extremely frustrating that we're being BILLED per minute that we have to wait on bitbucket to setup its own infrastructure.

Sometimes a step for me takes 15 minutes to "Setup" but only 1 minute to run. It feels extremely unjust to charge us for Atlassians own serverless issues. It can't be something we've done, as often that setup step will be instant.

We're using the Atlassian firebase deploy pipe for reference.

Our company already moved away from JIRA to another project management platform because it was just consistently extremely slow. It's looking more like we'll need to migrate off bitbucket too, given how slow merging branches and running pipelines has become in the last months. (Across more than 3 medium sized projects on different tech stacks).

It'd be nice if performance was a bigger priority at Atlassian.

Hello, how can this be neglected for so long? This is a crucial bottleneck in effectively using Bitbucket Pipelines. We are losing minutes every day and therefore hours every month because of this. Please give this ticket the attention it deserves?

Realy Attlassian? Realy...no reply, no help...nice support!

3 years and not even a single reply...?

0 votes

Same problem here... this definitely needs to be addressed.

0 votes
Bo Zheng I'm New Here Jul 08, 2021

Thank Atlassian, I am sitting right now in front of the laptop waiting for the pipeline and now the berlin time is nearly 20:00 in the evening. U can do nothing but wasting time 🤢

using default image 2, same issue

Any update?

I agree - my most recent pipeline was 4min & 41 seconds of which <50 seconds were tests that I want to run.

If i'm going to be continually billed for ~5x the amount of machine time I'm using, I'll just solve this problem with a githook and a bash script.

The variable time for build setup is ridiculous - sometimes it's 10 seconds, other times it's 10 minutes.

If I hadn't have already clocked over 50 build minutes and been charged overages I'd already be turning it off.

I know that the underlying architecture of pipelines is serverless, which makes me wonder if that's not the source of the slowness and variability...

We encounter same issue. Performance is really unstable - same pipeline might take 5 minutes in one run or 1 hour in another. Making me think if this was the right tool to use. 

We have the same issue. Steps are nice thing and it's probably useful in some cases that each step is a new docker image, but in our (and perhaps most?) use cases there is really no benefit and waiting 30s each step is not optimal.

0 votes

Conventional wisdom would provide resources to the environment. However, docker builds are conventionally slow. (Docker is not the rockstar, everyone thinks it is:)

You can optimize the build time using the --cache-from feature, see an example:

pipelines:
default:
- step:
services:
- docker
script:
- export IMAGE_NAME=your-docker-hub/your-repo:$BITBUCKET_BRANCH
- docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- docker pull $IMAGE_NAME || true
- docker build --cache-from $IMAGE_NAME -t $IMAGE_NAME .
- docker push $IMAGE_NAME

Avoid using the pipeline docker cache, only wastes time... :)

check https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html for more infos.

Thanks Danyal however this doesn't address the issue i'm looking at.

The setup you show and the docker cache only apply to when you're building docker images within your pipeline.

The image that takes forever to load for me is actually the custom build agent image. 

Like Jad Wahab likes this

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

New improvements to user management in Bitbucket Cloud 👥

Hey Community! We’re willing to wager that quite a few of you not only use Bitbucket, but administer it too. Our team is excited to share that we’ll be releasing improvements throughout this month of...

3,818 views 10 16
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you