[Open beta] Introducing Tests in Bitbucket Pipelines

[#1]

Hi Bitbucket community!

We’re thrilled to announce that Tests is now in open beta for Bitbucket Pipelines! This new feature for Bitbucket Cloud Standard and Premium workspaces introduces a single place inside your repository to:

• See the health of every test over time
• Drill into detailed execution history for each test
• Identify and quarantine flaky tests so they stop blocking your team
• Automatically skip quarantined tests in your pipelines’s build script to keep builds fast and reliable

All from within the place you already use to review code and powered by the JUnit-style XML reports you already generate today.

What is Tests?

Once enabled, you’ll see a new Tests tab in your Bitbucket repository.

This tab includes a summaries page that aggregates data for each individual test based on its recent runs, including:

• Failure rate – how often this test fails
• Average duration – how long it usually takes
• Variance – how much its execution time fluctuates

You can search and filter through the view to find a specific test, filter by labels, and filter by test state (such as whether the test is considered flaky or has been quarantined)

This makes it easy to spot frequently failing tests that hurt developer conference, slow tests that drag out pipelines, or unstable tests with highly variable run times.

You can also click into an individual test to see details about its execution, including:

• Build – the build that ran the test
• Commit – the commit associated with that run
• Last executed – when it last ran
• Execution duration – how long it took
• Outcome – whether it passed or failed

Take back control from flaky tests

With Tests, you can classify tests as problematic, specifically as: 

• Flaky – unstable and needing attention
• Quarantined – unreliable enough that it shouldn’t block builds right now

Behind the scenes, Bitbucket maintains a test metadata file for your repository that tracks these classifications. Your pipelines can download this file and use it to skip tests marked as 'quarantined.'

Check out the blog to see exactly how.

Try it now and share your feedback

If your Bitbucket Cloud workspace is on the Standard and Premium plan, you are automatically able to use these beta features by default.

1. Enable Bitbucket Pipelines for your repository (if it isn’t already).
2. Configure your test framework to generate JUnit-style XML reports or change the paths which are identified by the tests feature.
3. Open your repository and click the Tests tab.
4. Explore Test summaries and drill into Test executions to see your current test health.
5. Start marking flaky and quarantined tests to bring structure and clarity to your suite.
6. (Optional) Opt in to using the test metadata file in your pipeline steps so you can automatically skip quarantined tests and keep builds green and fast.

As we build out Tests, your feedback will help shape its future. Whether you have feedback, questions, or need help, this community thread is the go-to place to discuss your experience with Tests, report issues, or share ideas. Your input will directly influence the GA release and beyond, so let us know how Tests is working for you!

[Yuxin Wang]

Does it support child pipeline?

[Stijn Post]

We have set it all up according to the documentation (self hosted runner V4), and see a lot going right as is the junit xml is loaded well into the pipeline "Tests" tab but afterwards we see nothing going into the repo Test "Beta" tab.

Extract from build log:
Located test suite containing 4 tests, with 0 failures and 0 errors.
Located test suite containing 2 tests, with 0 failures and 0 errors.
Located test suite containing 2 tests, with 0 failures and 0 errors.
Located test suite containing 1 tests, with 0 failures and 0 errors.
Located test suite containing 6 tests, with 0 failures and 0 errors.
Located test suite containing 2 tests, with 0 failures and 0 errors.
Finished scanning for test reports. Found 1 test report files in 0 seconds.
Merged test suites, total number tests is 774, with 1 failures and 0 errors in 0 seconds.
Uploading test results
Finished uploading test results in 0 seconds.

[Gabriel]

Like @Stijn Post we have tests reporting success/failure in the pipeline but nothing is showing up in the new Tests (BETA) tab. Build log shows the test report getting picked up and processed.

Does the step have to run on the main development or production branches @Dan Hom? I was testing the junit changes to our repository from a feature branch to review the setup for this new capability. 

[Gummi A]

Same here as has been described, we see test results in Pipeline tab but the new Tests tab is empty. 

[Ben Ford]

Same issue as people above, works in pipelines. Nothing showing in the UI.

[Ben]

G'day all @Yuxin Wang @Stijn Post @Gabriel @Gummi A @Ben Ford !

Welcome to the Bitbucket Cloud community! Thank you for raising these concerns :)

Could you please create support tickets so that we can look into the specific issues for your repositories and engage the necessary engineering teams to assist further? You'll need to enter the workspace URL for your paid workspace and select Bitbucket Cloud as the product:

• https://support.atlassian.com/contact 

Below is some general information that I've received from our engineering team:

In the current release there are some rules which we have added for this feature to work:
1. You should be on Standard or Premium plans (I can see you are all on Premium)
2. As of now the test cases in `Tests` would show up only from the default branch (e.g main or master) pipelines, the feature branches tests wouldn't show up.
3. Tests executed in pipelines running Self hosted runners wouldn't show up. This support would come up in coming releases.
4. The Test reports which the frameworks are generating needs to be in the Junit xml format, more on this is here: https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/ 

Please check the documentation for more details - https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/

Cheers!

- Ben (Bitbucket Cloud Support)

[Rajkumar Singh]

@Yuxin Wang Yes the tests running in the child pipelines should show up in the Tests UI as long as the pipeline is fulfilling the criterrias  mentioned in this thread

[Stijn Post]

Ok that clears up the doubts, except for "only from the default branch" all fine. Would be great if we could set the tests to work with another branche like "development" @Ben 

[Ben Ford]

I agree with @Stijn Post , this should be for all branches. Or at least configurable for which branches. For me only showing it on the main branch is a little silly because if any tests failed they wouldn't get merged due to PR rules. So think it needs to change the way it is currently implemented.

[Stijn Post]

Indeed having it only in main/master doesnt make sense for us in our CI/CD pipeline. I would expect to be able to have this for one selectable branche

[Paulo Henrique dos Santos Eiterer]

We don't use Bitbucket Pipelines, but we have our own pipeline right now, and we use the Bitbucket API to send the build status. It would be great if we could still use our own pipeline and use the API to post the JUnit-style XML reports. @Ben

[Rajkumar Singh]

Thanks so much for the detailed feedback, @Stijn Post @Ben Ford  – this is really helpful context.

You’re right that limiting this to the default branch isn’t ideal for many CI/CD setups, especially where you’re enforcing PR rules and doing most of your validation on branches like development before anything reaches main/master.

Right now, we’ve released this version of the feature as an experiment so we can validate the core experience, learn from real usage, and iterate quickly. Branch flexibility is one areas we will be looking into as we get more feedback about the feature and improve it.

In the meantime, we’d really appreciate it if you could try the current version and share how it fits (or doesn’t fit) into your workflow: 

• Are there any rules or patterns you use today to decide where tests should run or be surfaced?

• Highlight any other functionality that you think would be useful for your workflows?

Feel free to raise support ticket of a functionality request and we would be happy to evolve this new feature into something which is improving all the aspects related to your needs.

[Rajkumar Singh]

@Paulo Henrique dos Santos Eiterer 

Thanks a lot for sharing this, and for already integrating with Bitbucket via the build status API – that’s super helpful context.

Right now, posting JUnit-style XML reports directly via the Bitbucket API (without using Bitbucket Pipelines) isn’t supported. The current version of test management is coupled to Pipelines so we can make sure we fully understand usage and scale before we open it up further.

That said, your workflow is exactly the kind of use case we’d like to support as we mature the feature. To do that well, we need to understand more about how you’re running tests and what “good” would look like for you:

• What CI system are you using today?

• How are you currently generating and storing your JUnit XML reports?

• At what point in your pipeline would you ideally send those reports to Bitbucket?

If you’re open to it, we’d love to set up a short call to go deeper into your setup and requirements so we can factor this into our roadmap. If that works for you, please let us know your preferred times/time zone and we’ll reach out to schedule something.

[Stijn Post]

@Rajkumar Singh Thanks for the update. We wont be testing this feature in main/master as that process just doesnt fit our CI/CD.

So until we can set the branche where to run it all from we will continue to use https://testomat.io as here we can have our e2e and unit test suite run on every PR independent of branche and get nice historic insights in test runs

[Rajkumar Singh]

@Stijn Post thanks for the extra context and for sharing how you’re using testomat.io today – that’s really helpful.

We’ve created a feature request to better support your use case:
https://jira.atlassian.com/browse/BCLOUD-23991

We’ll use this ticket to gather additional interest from other customers with similar CI/CD setups and will explore how we can support configuring which branch these tests should run from. I’ve linked your feedback there so it’s part of the conversation as we evaluate next steps.

[Andrew Simpson]

This is a good feature, but: one of the most important metrics when testing is how much of the code is actually exercised by tests, if it's trending in the right direction, how PRs affect code coverage, etc. I'd really prefer to have visualisation of code coverage data rather than detection of flaky tests with "statistical algorithms and AI agents".

If AI agents really must be jammed in somewhere, code coverage integration would allow detection of untested areas of code, which an AI agent could automatically write a test to cover.

At the very least, an update on the open code coverage feature request would be nice.