Quarantined tests are not written into BITBUCKET_TEST_METADATA_FILE_PATH

Ivan Litskevich

Hello, we're having an issue where the quarantined tests are not appearing in /opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/test-case/meta-data.json (the file itself in not present on build machines), which causes quarantined tests to be executed during pipeline runs. Our setup:

1. Running against master

2. master is the only branch being tracked under Bitbucket Tests

3. Pipelines have

test-management:
                inject-classification: true

4. We're running Playwright tests and followed instructions from https://support.atlassian.com/bitbucket-cloud/docs/quarantining-flaky-tests/ to get it setup

5. Have a quarantined test in Bitbucket Tests

6. Pipelines are running on self-hosted runners v5

Any suggestions what else needs to be done to make it work?

Evgenii

Hi @Ivan Litskevich,

That's the self-hosted runners. Test metadata injection doesn't support them yet, Atlassian confirmed this in the Tests beta announcement thread: support for self-hosted runners "isn't available just yet, but it's definitely on our roadmap", and they ask affected users to try the feature on Atlassian-hosted runners for now. So your config is most likely fine, the file just never gets injected on your v5 runners.

Atlassian wrote about it here, look in comments: https://community.atlassian.com/forums/Bitbucket-articles/Open-beta-Introducing-Tests-in-Bitbucket-Pipelines/ba-p/3181650

You can confirm it in one run: execute the same step without runs-on on Atlassian-hosted infrastructure. The file at BITBucket_TEST_METADATA_FILE_PATH should appear there with your quarantined test in it.

Until runner support ships, the options are either running the test steps on cloud runners, or keeping self-hosted and skipping quarantined tests by your own list (a file in the repo, or a repository variable) instead of the injected metadata. Worth dropping a comment in that announcement thread too, they're collecting exactly this kind of beta feedback and it may bump self-hosted support up the roadmap.