Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

BItbucket pipelines suddenly becoming very slow today

Emmanuel Chavane April 26, 2022

I have been using pipelines for several weeks with builds that usually last around 6 minutes. Today they all jumped to at least 20 minutes, and I just killed one that was not completed after 45 minutes.

Looking at the logs, it seems that the "Build setup" phase is the culprit, used most of the time. Caches are apparently downloaded at normal speed (the only timing information I can read in the phase log).

I see no incident reported on the Bitbucket status page.

So I am not sure where to go from here. It is the first time we have had such an incident, but it's kinda blocking for the team (and our time budget).

Copying Build phase log below in case someone sees something interesting/meaningful.

Thanks

 

Build setup14m 55s
+ umask 000
+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="features/mia-algolia-sync" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR
Cloning into '/opt/atlassian/pipelines/agent/build'...
+ git reset --hard 362cec8d48b956ba40c020fc0046b9c31b2c386d
HEAD is now at 362cec8  [MIA] - ...
+ git config user.name bitbucket-pipelines
+ git config user.email commits-noreply@bitbucket.org
+ git config push.default current
+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/
+ git reflog expire --expire=all --all
+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude
+ chmod 777 $BUILD_DIR
Cache "maven": Downloading
Cache "maven": Downloaded 214.3 MiB in 6 seconds
Cache "maven": Extracting
Cache "maven": Extracted in 1 seconds
Images used:
./mvnw clean test4m 9s
Build teardown

 

5 answers

1 accepted

0 votes
Answer accepted
Patrik S
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 27, 2022

Hi @Emmanuel Chavane

Thank you for reporting this issue to us.

Based on our investigation, we've identified that images hosted in Google Container Registry (gcr.io) are slow to pull, whilst images hosted on Artifact Registry  (pkg.dev) are not impacted.

Our engineering team is already in contact with Google as we suspect that Pipelines is possibly being rate-limited/throtled by Google which impacts users who are using GCR images on Pipelines.

You can refer to our status page for further updates. - https://bitbucket.status.atlassian.com/incidents/4gfrv6xyym2t

Also, since Artifact registry is recommended over Container registry by Google (See this link), we recommend transitioning to the Artifact registry as a workaround while we're working with Google on this issue.

Thank you @Emmanuel Chavane .

Kind regards,

Patrik S

Emmanuel Chavane April 27, 2022

Thanks for the reply @Patrik S . Very reassuring to see you guys are monitoring this forum and are able to provide detailed explanations / workarounds :)

2 votes
Emmanuel Chavane April 27, 2022

The problem is still here today: absolutely random durations for the build setup phase (from 1 minute to 50 minutes).

Found this thread https://community.atlassian.com/t5/Bitbucket-questions/Pipelines-Build-Setup-taking-too-long/qaq-p/857558 that describes the same problem and references a ticket, with apparently no solution.

Not sure if someone from Atlassian is reading this, but as a paying customer, watching the numbers of minutes that are used for a build phase (on which I have no control) multiplied by 10, with no way to understand if this is temporary or not, is just not acceptable (in addition to the impact on the dev team productivity).

I have been happy with pipelines for the few weeks we have used it until yesterday, but this issue will be a killer if there is no explanation/resolution very quickly. Will need to move to another CI solution. 

David Golden April 27, 2022

FWIW We've been using Bitbucket for our pipelines for a couple years and this has never happened to us before to my knowledge.

Like Emmanuel Chavane likes this
2 votes
David Golden April 26, 2022

Also seeing this. A pipeline that's always been 16-18mins just took 41min.

0 votes
Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 8, 2022

Hi @Nelu Donelly Akejellu@David Golden, and @Emmanuel Chavane,

If you're still experiencing issues, please create a ticket with the support team so we can look into your respective issues.

You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

Please make sure to share in the ticket
- the URL of a build that is taking now more time than previously
- the URL of a previous build that was taking less time

This way we can check the build logs and extra logs from our side in order to further investigate.

The support ticket will be visible only to the reporter and Atlassian staff, so any build URLs and other private info you post there will not be publicly visible.

Kind regards,
Theodora

0 votes
Nelu Donelly Akejellu August 5, 2022

Hello @Patrik S  I also have the same issue. A build that takes less than 5 minutes now takes over 120 minutes and it times out. How can this issue be resolved ?

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events