How to compute Test Failure Rate for each test automatically

Vrajesh Dave March 20, 2023

Hello!

I use XRAY server and would like to compute Test Failure rate for each of my tests automatically and on regular basis.

Test failure rate = (No. of times test failed) / (Total no. of times test got executed)

I can get a similar statistics from Squish Test Center. Please see image below:

 

FailureRate_Squish.png

 

I have already tried following:

1. I figured out that I can compute Rate=A/B manually for a specific test (e.g. DEMO-9) using the following JQL

A = [issuetype = 'Test Execution' and issue in testTestExecutions('DEMO-9', 'FAIL')]
B = [issuetype = 'Test Execution' and (issue in testTestExecutions('DEMO-9', 'FAIL') OR issue in testTestExecutions('DEMO-9', 'PASS')]

For this, I referred documentation at the link below:

https://docs.getxray.app/display/XRAY/Enhanced+querying+with+JQL

But there is no way that I could find to do this automatically for all of my tests.

2. I tried but could not accomplish this with Rich Filters on my Dashboard to show list of my tests with a column for "Failure rate".

3. I was not able to find any column on the default XRay Reports or Test panel to get such a statistics. A similar statistics is available on Test Execution Report for "Success Rate" (please see image below) but that is applicable for the entire execution which may contain several tests. I would like to see it for individual tests.

TestExecutionReport.png

Any help is greatly appreciated. Thanks!

1 answer

0 votes
Stefan Salzl
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
March 24, 2023

Hi @Vrajesh Dave - welcome to the community

Do you organize your test cases/test phases with TestPlans? This would give you the possibility to automativally get metrics for testcases in your test plan:

https://docs.getxray.app/display/XRAYCLOUD/Test+Plans+Metrics+Report

 

Please also see the following link to get an overview of good practices and ways to organize your tests in XRAY:
https://docs.getxray.app/display/XRAY/Terms+and+Concepts
https://docs.getxray.app/display/XRAY/Organizing+Tests
https://docs.getxray.app/display/XRAY/Planning+Tests

 

Hope this was helpful.

Best
Stefan

Vrajesh Dave March 24, 2023

Hi @Stefan Salzl, Thank you for the warm welcome! I'm excited to be a part of this community.

Yes, we do organize our test cases with Test Plans. I am aware of the general concepts and we do follow most of the good practices. Thanks for the links!

I was not able to get an answer to my specific question from the links though.

Thank you so much for your reply!

Best Regards,

Vrajesh

Stefan Salzl
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
March 25, 2023

Hi @Vrajesh Dave 

one of the links leads to the Tesplan Metrics which provide Test success rate in the report.

As you organize your test cases in testplans this should be possible with the mentioned link.

In case I got something wrong with your requirement please appologize. May I kindly ask then for further specification in order to try to find an applicable solution.

Best
Stefan

Vrajesh Dave March 26, 2023

Hi @Stefan Salzl , thanks for your reply!

I believe you are talking about the documentation as shown in the image below:

Help_TestPlan_SuccessRate.png

 

I can see this metric in our Test Plan as below:

TestPlanReport.png

 

 

Unfortunately, this is not the number I am looking for. This Success rate is for the entire test plan and I am guessing it is = (Number of Passed Test) / (Total number of tests in the test plan that got executed)

The number that I am looking for is individual Test failure rate as described in my original query: Test failure rate = (No. of times a particular test failed) / (Total no. of times that particular test got executed). Here this particular test could have been executed in any number of test plans and considers the entire history of its execution.

I hope that this gives a clear picture of my requirement. Thanks for your time!

Best Regards,

Vrajesh

Mathias Kristensen
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
May 14, 2024

Hello @Vrajesh Dave 

Did you ever figure out the answer to this one? I have been looking for something similar, but does not seem to be possible

Best Regards

Mathias

Vrajesh Dave May 14, 2024

Hello @Mathias Kristensen , no I have not found an answer to this question yet.

Vrajesh Dave May 14, 2024

Hello @Mathias Kristensen , I got to know that this improvement is reported to Xray team via: XRAYCLOUD-4229 and XRAYCLOUD-1662 
You may choose to Vote so that they can keep track of our interest and Watch to receive news on the improvement.

Looks like the above improvements are not exactly the same as required. I will post update here in case I find something related. Thanks!

Vrajesh Dave May 16, 2024

Updates: I checked with Tech support from Xray. They have logged an enhancement request: XRAYCLOUD-7175 

You may choose to Vote so that they can keep track of our interest and Watch to receive news on the improvement.

Like Mathias Kristensen likes this
Mathias Kristensen
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
May 20, 2024

Thanks!
I found a way to do it manually by exporting the numbers with xray document generator, and then compute it. But that is a bit cumbersome.

Like # people like this

Suggest an answer

Log in or Sign up to answer