[Open beta] Introducing Tests in Bitbucket Pipelines

[#1]

Hi Bitbucket community!

We’re thrilled to announce that Tests is now in open beta for Bitbucket Pipelines! This new feature for Bitbucket Cloud Standard and Premium workspaces introduces a single place inside your repository to:

• See the health of every test over time
• Drill into detailed execution history for each test
• Identify and quarantine flaky tests so they stop blocking your team
• Automatically skip quarantined tests in your pipelines’s build script to keep builds fast and reliable

All from within the place you already use to review code and powered by the JUnit-style XML reports you already generate today.

What is Tests?

Once enabled, you’ll see a new Tests tab in your Bitbucket repository.

This tab includes a summaries page that aggregates data for each individual test based on its recent runs, including:

• Failure rate – how often this test fails
• Average duration – how long it usually takes
• Variance – how much its execution time fluctuates

You can search and filter through the view to find a specific test, filter by labels, and filter by test state (such as whether the test is considered flaky or has been quarantined)

This makes it easy to spot frequently failing tests that hurt developer conference, slow tests that drag out pipelines, or unstable tests with highly variable run times.

You can also click into an individual test to see details about its execution, including:

• Build – the build that ran the test
• Commit – the commit associated with that run
• Last executed – when it last ran
• Execution duration – how long it took
• Outcome – whether it passed or failed

Take back control from flaky tests

With Tests, you can classify tests as problematic, specifically as: 

• Flaky – unstable and needing attention
• Quarantined – unreliable enough that it shouldn’t block builds right now

Behind the scenes, Bitbucket maintains a test metadata file for your repository that tracks these classifications. Your pipelines can download this file and use it to skip tests marked as 'quarantined.'

Check out the blog to see exactly how.

Try it now and share your feedback

If your Bitbucket Cloud workspace is on the Standard and Premium plan, you are automatically able to use these beta features by default.

1. Enable Bitbucket Pipelines for your repository (if it isn’t already).
2. Configure your test framework to generate JUnit-style XML reports or change the paths which are identified by the tests feature.
3. Open your repository and click the Tests tab.
4. Explore Test summaries and drill into Test executions to see your current test health.
5. Start marking flaky and quarantined tests to bring structure and clarity to your suite.
6. (Optional) Opt in to using the test metadata file in your pipeline steps so you can automatically skip quarantined tests and keep builds green and fast.

As we build out Tests, your feedback will help shape its future. Whether you have feedback, questions, or need help, this community thread is the go-to place to discuss your experience with Tests, report issues, or share ideas. Your input will directly influence the GA release and beyond, so let us know how Tests is working for you!

[Yuxin Wang]

Does it support child pipeline?

[Stijn Post]

We have set it all up according to the documentation (self hosted runner V4), and see a lot going right as is the junit xml is loaded well into the pipeline "Tests" tab but afterwards we see nothing going into the repo Test "Beta" tab.

Extract from build log:
Located test suite containing 4 tests, with 0 failures and 0 errors.
Located test suite containing 2 tests, with 0 failures and 0 errors.
Located test suite containing 2 tests, with 0 failures and 0 errors.
Located test suite containing 1 tests, with 0 failures and 0 errors.
Located test suite containing 6 tests, with 0 failures and 0 errors.
Located test suite containing 2 tests, with 0 failures and 0 errors.
Finished scanning for test reports. Found 1 test report files in 0 seconds.
Merged test suites, total number tests is 774, with 1 failures and 0 errors in 0 seconds.
Uploading test results
Finished uploading test results in 0 seconds.

[Gabriel]

Like @Stijn Post we have tests reporting success/failure in the pipeline but nothing is showing up in the new Tests (BETA) tab. Build log shows the test report getting picked up and processed.

Does the step have to run on the main development or production branches @Dan Hom? I was testing the junit changes to our repository from a feature branch to review the setup for this new capability. 

[Gummi A]

Same here as has been described, we see test results in Pipeline tab but the new Tests tab is empty. 

[Ben Ford]

Same issue as people above, works in pipelines. Nothing showing in the UI.

[Ben]

G'day all @Yuxin Wang @Stijn Post @Gabriel @Gummi A @Ben Ford !

Welcome to the Bitbucket Cloud community! Thank you for raising these concerns :)

Could you please create support tickets so that we can look into the specific issues for your repositories and engage the necessary engineering teams to assist further? You'll need to enter the workspace URL for your paid workspace and select Bitbucket Cloud as the product:

• https://support.atlassian.com/contact 

Below is some general information that I've received from our engineering team:

In the current release there are some rules which we have added for this feature to work:
1. You should be on Standard or Premium plans (I can see you are all on Premium)
2. As of now the test cases in `Tests` would show up only from the default branch (e.g main or master) pipelines, the feature branches tests wouldn't show up.
3. Tests executed in pipelines running Self hosted runners wouldn't show up. This support would come up in coming releases.
4. The Test reports which the frameworks are generating needs to be in the Junit xml format, more on this is here: https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/ 

Please check the documentation for more details - https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/

Cheers!

- Ben (Bitbucket Cloud Support)

[Rajkumar Singh]

@Yuxin Wang Yes the tests running in the child pipelines should show up in the Tests UI as long as the pipeline is fulfilling the criterrias  mentioned in this thread

[Stijn Post]

Ok that clears up the doubts, except for "only from the default branch" all fine. Would be great if we could set the tests to work with another branche like "development" @Ben 

[Ben Ford]

I agree with @Stijn Post , this should be for all branches. Or at least configurable for which branches. For me only showing it on the main branch is a little silly because if any tests failed they wouldn't get merged due to PR rules. So think it needs to change the way it is currently implemented.

[Stijn Post]

Indeed having it only in main/master doesnt make sense for us in our CI/CD pipeline. I would expect to be able to have this for one selectable branche