Xray Test Execution Success Rate

Nancy Leaver

How is the success rate calculated in Xray? For a test execution, we have 1795 tests, of which 1577 are marked as PASS. If the success rate is calculated at 1577/1795 that would be 88%, but the report shows it at 93%. This is higher than our actual progress of 90% which seems impossible. 

Humashankar VJ

Hi @Nancy Leaver 

Xray provides administrators with the flexibility to customize test statuses that are considered "Passed" or "Failed" for reporting purposes.

• Excluding additional statuses, such as Blocked, Not Applicable, or To Do, from the calculation can significantly impact the results.
• By reducing the total number of tests considered, the success rate may appear artificially inflated. For instance, if 1,795 tests are initially counted, but Xray excludes certain tests, the actual number used for calculation may be lower, leading to a potentially weighted success rate.

It's essential to note that Xray's complexity and variability in configuration and reporting methods can influence the accuracy of these calculations.

For instance:

• Total tests: 1795
• Passed tests: 1577
• Success rate expected: 1577 / 1795 = 88%

But if Xray excludes 100 tests with statuses like Blocked or To Do, the denominator becomes 1695, and the success rate would be:

• Success rate after exclusion: 1577 / 1695 = 93%

Hope this helps - Happy to help further!!
Thank you very much and have a great one!
Warm regards