Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,463,474
Community Members
 
Community Events
176
Community Groups

BItbucket pipelines suddenly becoming very slow today

I have been using pipelines for several weeks with builds that usually last around 6 minutes. Today they all jumped to at least 20 minutes, and I just killed one that was not completed after 45 minutes.

Looking at the logs, it seems that the "Build setup" phase is the culprit, used most of the time. Caches are apparently downloaded at normal speed (the only timing information I can read in the phase log).

I see no incident reported on the Bitbucket status page.

So I am not sure where to go from here. It is the first time we have had such an incident, but it's kinda blocking for the team (and our time budget).

Copying Build phase log below in case someone sees something interesting/meaningful.

Thanks

 

Build setup14m 55s
+ umask 000
+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="features/mia-algolia-sync" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR
Cloning into '/opt/atlassian/pipelines/agent/build'...
+ git reset --hard 362cec8d48b956ba40c020fc0046b9c31b2c386d
HEAD is now at 362cec8  [MIA] - ...
+ git config user.name bitbucket-pipelines
+ git config user.email commits-noreply@bitbucket.org
+ git config push.default current
+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/
+ git reflog expire --expire=all --all
+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude
+ chmod 777 $BUILD_DIR
Cache "maven": Downloading
Cache "maven": Downloaded 214.3 MiB in 6 seconds
Cache "maven": Extracting
Cache "maven": Extracted in 1 seconds
Images used:
./mvnw clean test4m 9s
Build teardown

 

5 answers

1 accepted

0 votes
Answer accepted
Patrik S Atlassian Team Apr 27, 2022

Hi @Emmanuel Chavane

Thank you for reporting this issue to us.

Based on our investigation, we've identified that images hosted in Google Container Registry (gcr.io) are slow to pull, whilst images hosted on Artifact Registry  (pkg.dev) are not impacted.

Our engineering team is already in contact with Google as we suspect that Pipelines is possibly being rate-limited/throtled by Google which impacts users who are using GCR images on Pipelines.

You can refer to our status page for further updates. - https://bitbucket.status.atlassian.com/incidents/4gfrv6xyym2t

Also, since Artifact registry is recommended over Container registry by Google (See this link), we recommend transitioning to the Artifact registry as a workaround while we're working with Google on this issue.

Thank you @Emmanuel Chavane .

Kind regards,

Patrik S

Thanks for the reply @Patrik S . Very reassuring to see you guys are monitoring this forum and are able to provide detailed explanations / workarounds :)

The problem is still here today: absolutely random durations for the build setup phase (from 1 minute to 50 minutes).

Found this thread https://community.atlassian.com/t5/Bitbucket-questions/Pipelines-Build-Setup-taking-too-long/qaq-p/857558 that describes the same problem and references a ticket, with apparently no solution.

Not sure if someone from Atlassian is reading this, but as a paying customer, watching the numbers of minutes that are used for a build phase (on which I have no control) multiplied by 10, with no way to understand if this is temporary or not, is just not acceptable (in addition to the impact on the dev team productivity).

I have been happy with pipelines for the few weeks we have used it until yesterday, but this issue will be a killer if there is no explanation/resolution very quickly. Will need to move to another CI solution. 

FWIW We've been using Bitbucket for our pipelines for a couple years and this has never happened to us before to my knowledge.

Like Emmanuel Chavane likes this

Also seeing this. A pipeline that's always been 16-18mins just took 41min.

0 votes

Hi @Nelu Donelly Akejellu@David Golden, and @Emmanuel Chavane,

If you're still experiencing issues, please create a ticket with the support team so we can look into your respective issues.

You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

Please make sure to share in the ticket
- the URL of a build that is taking now more time than previously
- the URL of a previous build that was taking less time

This way we can check the build logs and extra logs from our side in order to further investigate.

The support ticket will be visible only to the reporter and Atlassian staff, so any build URLs and other private info you post there will not be publicly visible.

Kind regards,
Theodora

Hello @Patrik S  I also have the same issue. A build that takes less than 5 minutes now takes over 120 minutes and it times out. How can this issue be resolved ?

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events