Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Pipelines Build Setup taking too long Edited

I've been playing around with pipelines and have managed to get my pipeline working exactly the way I want. However it takes 5 minutes to run and looking back through the logs most of this is due to the Build Setup portion of each step.

My pipeline has 3 steps and the Build Setup takes between 30s and 90s on each step. 

I was using a private docker image but have changed this to be a public image as I understand these are cached however this hasn't affected performance.  The image is large (650MB). The repo is about 25mb.

Any ideas of what I can change to improve / diagnose the performance issues?

 

image.png

image:
name: pjcable/temp:meteor-base-v3
#username: pjcable
#email: paul@highspecsolutions.com
#password: $DOCKER_HUB_PASSWORD

options:
max-time: 5

pipelines:
branches:
master:

- step:
name: Testing
max-time: 3
caches:
- composer
script:
- cp .env.example .env
- composer install
- php artisan key:generate
- sleep 10
- vendor/bin/phpunit
- php artisan serve &
- php artisan dusk
- kill $(jobs -p)
services:
- mysql
- dusk-runner

- step:
name: Generate production assets
caches:
- node
script:
- echo "<?php return [ 'number' => ${BITBUCKET_BUILD_NUMBER}];" > config/version.php
- npm install
- npm run production
- git config --global user.name "Meteor Bot"
- git config --global user.email "paul+meteor-bot@highspecsolutions.com"
- git commit -am "Committing V${BITBUCKET_BUILD_NUMBER} production assets"
- git tag "V${BITBUCKET_BUILD_NUMBER}"
- git remote remove origin
- git remote add origin https://${BB_AUTH_STRING}@bitbucket.org/meteorsg/web.git
- git push origin "V${BITBUCKET_BUILD_NUMBER}"
- step:
name: Deploy to staging
deployment: staging
#trigger: manual
script:
- echo "Push to staging server"
- ssh -v bitbucket@meteor.highspecsolutions.com "echo hello"
- envoy run deploy-staging --verbose --release_tag=V${BITBUCKET_BUILD_NUMBER}

definitions:
services:
mysql:
image: mysql:5.7
environment:
MYSQL_DATABASE: 'homestead'
MYSQL_USER: 'homestead'
MYSQL_PASSWORD: 'secret'
MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
dusk-runner:
image: pjcable/dusk-runner:latest

 

 

11 answers

We have the same issue. Pipeline's build setup speed varies from 30 seconds to 50 minutes!

This is the output of the build setup and unfortunately there is no timing being printed out in details except the time took for downloading caches. However, the whole step took 46 minutes:

 

+ umask 000+ GIT_LFS_SKIP_SMUDGE=1 git clone --branch="CT-123-change-parameter-name-in-entity" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/abcd/abcd.git $BUILD_DIR ; git reset --hard 8bdfe2cf4dd33902b321208d92a90f8f236d3660 ; git remote set-url origin git@bitbucket.org:abcd/abcd.git
Cloning into '/opt/atlassian/pipelines/agent/build'...
HEAD is now at 8bdfe2cf4 CT-123: issue description
+ chmod 777 $BUILD_DIR
Cache "maven": Downloading
Cache "maven": Downloaded 226 MB in 14 seconds
Cache "maven": Extracting
Cache "maven": Extracted in 3 seconds
Cache "nodemodules": Downloading
Cache "nodemodules": Downloaded 4 MB in 1 seconds
Cache "nodemodules": Extracting
Cache "nodemodules": Extracted in 0 seconds

this not only costs us but also delays the development team. Any idea?

 

 image.png

Any update on this question?  

I'm raising a task. https://jira.atlassian.com/browse/BCLOUD-19451

Yes, this issue is starting to become fairly frustrating for me also.

I have the same issue. We are a paid customer and sometimes the Build setup step at the very beginning takes 2+ minutes.

0 votes

Conventional wisdom would provide resources to the environment. However, docker builds are conventionally slow. (Docker is not the rockstar, everyone thinks it is:)

You can optimize the build time using the --cache-from feature, see an example:

pipelines:
default:
- step:
services:
- docker
script:
- export IMAGE_NAME=your-docker-hub/your-repo:$BITBUCKET_BRANCH
- docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- docker pull $IMAGE_NAME || true
- docker build --cache-from $IMAGE_NAME -t $IMAGE_NAME .
- docker push $IMAGE_NAME

Avoid using the pipeline docker cache, only wastes time... :)

check https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html for more infos.

Thanks Danyal however this doesn't address the issue i'm looking at.

The setup you show and the docker cache only apply to when you're building docker images within your pipeline.

The image that takes forever to load for me is actually the custom build agent image. 

Like Jad Wahab likes this

We have the same issue. Steps are nice thing and it's probably useful in some cases that each step is a new docker image, but in our (and perhaps most?) use cases there is really no benefit and waiting 30s each step is not optimal.

We encounter same issue. Performance is really unstable - same pipeline might take 5 minutes in one run or 1 hour in another. Making me think if this was the right tool to use. 

I agree - my most recent pipeline was 4min & 41 seconds of which <50 seconds were tests that I want to run.

If i'm going to be continually billed for ~5x the amount of machine time I'm using, I'll just solve this problem with a githook and a bash script.

The variable time for build setup is ridiculous - sometimes it's 10 seconds, other times it's 10 minutes.

If I hadn't have already clocked over 50 build minutes and been charged overages I'd already be turning it off.

I know that the underlying architecture of pipelines is serverless, which makes me wonder if that's not the source of the slowness and variability...

Any update?

0 votes

using default image 2, same issue

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Calling any interview participants for Bitbucket Data Center

Hi everyone,  We are looking to learn more about development teams’ workflows and pain points, especially around DevOps, integrations, administration, scale, security, and the related challeng...

441 views 4 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you