BItbucket pipelines suddenly becoming very slow today

[Emmanuel Chavane]

I have been using pipelines for several weeks with builds that usually last around 6 minutes. Today they all jumped to at least 20 minutes, and I just killed one that was not completed after 45 minutes.

Looking at the logs, it seems that the "Build setup" phase is the culprit, used most of the time. Caches are apparently downloaded at normal speed (the only timing information I can read in the phase log).

I see no incident reported on the Bitbucket status page.

So I am not sure where to go from here. It is the first time we have had such an incident, but it's kinda blocking for the team (and our time budget).

Copying Build phase log below in case someone sees something interesting/meaningful.

Thanks

Build setup14m 55s

+ umask 000

+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="features/mia-algolia-sync" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR

Cloning into '/opt/atlassian/pipelines/agent/build'...

+ git reset --hard 362cec8d48b956ba40c020fc0046b9c31b2c386d

HEAD is now at 362cec8  [MIA] - ...

+ git config user.name bitbucket-pipelines

+ git config user.email commits-noreply@bitbucket.org

+ git config push.default current

+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/

+ git remote set-url origin http://bitbucket.org/$BITBUCKET_REPO_FULL_NAME

+ git reflog expire --expire=all --all

+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude

+ chmod 777 $BUILD_DIR

Cache "maven": Downloading

Cache "maven": Downloaded 214.3 MiB in 6 seconds

Cache "maven": Extracting

Cache "maven": Extracted in 1 seconds

Images used:

build : eu.gcr.io/janngo-studio-shared/build-machine@sha256:b5692d9eb0362a80a63309cc9933b97432749f24b82310bfbbf995f5c5063fdd

./mvnw clean test4m 9s

Build teardown

[Patrik S · Accepted solution]

Hi @Emmanuel Chavane , 

Thank you for reporting this issue to us.

Based on our investigation, we've identified that images hosted in Google Container Registry (gcr.io) are slow to pull, whilst images hosted on Artifact Registry  (pkg.dev) are not impacted.

Our engineering team is already in contact with Google as we suspect that Pipelines is possibly being rate-limited/throtled by Google which impacts users who are using GCR images on Pipelines.

You can refer to our status page for further updates. - https://bitbucket.status.atlassian.com/incidents/4gfrv6xyym2t

Also, since Artifact registry is recommended over Container registry by Google (See this link), we recommend transitioning to the Artifact registry as a workaround while we're working with Google on this issue.

Thank you @Emmanuel Chavane .

Kind regards,

Patrik S

[Emmanuel Chavane]

Thanks for the reply @Patrik S . Very reassuring to see you guys are monitoring this forum and are able to provide detailed explanations / workarounds :)

[Emmanuel Chavane]

The problem is still here today: absolutely random durations for the build setup phase (from 1 minute to 50 minutes).

Found this thread https://community.atlassian.com/t5/Bitbucket-questions/Pipelines-Build-Setup-taking-too-long/qaq-p/857558 that describes the same problem and references a ticket, with apparently no solution.

Not sure if someone from Atlassian is reading this, but as a paying customer, watching the numbers of minutes that are used for a build phase (on which I have no control) multiplied by 10, with no way to understand if this is temporary or not, is just not acceptable (in addition to the impact on the dev team productivity).

I have been happy with pipelines for the few weeks we have used it until yesterday, but this issue will be a killer if there is no explanation/resolution very quickly. Will need to move to another CI solution. 

[David Golden]

FWIW We've been using Bitbucket for our pipelines for a couple years and this has never happened to us before to my knowledge.

[David Golden]

Also seeing this. A pipeline that's always been 16-18mins just took 41min.

[Theodora Boudale]

Hi @Nelu Donelly Akejellu, @David Golden, and @Emmanuel Chavane,

If you're still experiencing issues, please create a ticket with the support team so we can look into your respective issues.

You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

Please make sure to share in the ticket
- the URL of a build that is taking now more time than previously
- the URL of a previous build that was taking less time

This way we can check the build logs and extra logs from our side in order to further investigate.

The support ticket will be visible only to the reporter and Atlassian staff, so any build URLs and other private info you post there will not be publicly visible.

Kind regards,
Theodora

[Nelu Donelly Akejellu]

Hello @Patrik S  I also have the same issue. A build that takes less than 5 minutes now takes over 120 minutes and it times out. How can this issue be resolved ?