How to compute Test Failure Rate for each test automatically

[Vrajesh Dave]

Hello!

I use XRAY server and would like to compute Test Failure rate for each of my tests automatically and on regular basis.

Test failure rate = (No. of times test failed) / (Total no. of times test got executed)

I can get a similar statistics from Squish Test Center. Please see image below:

I have already tried following:

1. I figured out that I can compute Rate=A/B manually for a specific test (e.g. DEMO-9) using the following JQL

A = [issuetype = 'Test Execution' and issue in testTestExecutions('DEMO-9', 'FAIL')]
B = [issuetype = 'Test Execution' and (issue in testTestExecutions('DEMO-9', 'FAIL') OR issue in testTestExecutions('DEMO-9', 'PASS')]

For this, I referred documentation at the link below:

https://docs.getxray.app/display/XRAY/Enhanced+querying+with+JQL

But there is no way that I could find to do this automatically for all of my tests.

2. I tried but could not accomplish this with Rich Filters on my Dashboard to show list of my tests with a column for "Failure rate".

3. I was not able to find any column on the default XRay Reports or Test panel to get such a statistics. A similar statistics is available on Test Execution Report for "Success Rate" (please see image below) but that is applicable for the entire execution which may contain several tests. I would like to see it for individual tests.

Any help is greatly appreciated. Thanks!

[Stefan Salzl]

Hi @Vrajesh Dave - welcome to the community

Do you organize your test cases/test phases with TestPlans? This would give you the possibility to automativally get metrics for testcases in your test plan:

https://docs.getxray.app/display/XRAYCLOUD/Test+Plans+Metrics+Report

Please also see the following link to get an overview of good practices and ways to organize your tests in XRAY:
https://docs.getxray.app/display/XRAY/Terms+and+Concepts
https://docs.getxray.app/display/XRAY/Organizing+Tests
https://docs.getxray.app/display/XRAY/Planning+Tests

Hope this was helpful.

Best
Stefan

[Vrajesh Dave]

Hi @Stefan Salzl, Thank you for the warm welcome! I'm excited to be a part of this community.

Yes, we do organize our test cases with Test Plans. I am aware of the general concepts and we do follow most of the good practices. Thanks for the links!

I was not able to get an answer to my specific question from the links though.

Thank you so much for your reply!

Best Regards,

Vrajesh

[Stefan Salzl]

Hi @Vrajesh Dave 

one of the links leads to the Tesplan Metrics which provide Test success rate in the report.

As you organize your test cases in testplans this should be possible with the mentioned link.

In case I got something wrong with your requirement please appologize. May I kindly ask then for further specification in order to try to find an applicable solution.

Best
Stefan

[Vrajesh Dave]

Hi @Stefan Salzl , thanks for your reply!

I believe you are talking about the documentation as shown in the image below:

I can see this metric in our Test Plan as below:

Unfortunately, this is not the number I am looking for. This Success rate is for the entire test plan and I am guessing it is = (Number of Passed Test) / (Total number of tests in the test plan that got executed)

The number that I am looking for is individual Test failure rate as described in my original query: Test failure rate = (No. of times a particular test failed) / (Total no. of times that particular test got executed). Here this particular test could have been executed in any number of test plans and considers the entire history of its execution.

I hope that this gives a clear picture of my requirement. Thanks for your time!

Best Regards,

Vrajesh