Hello!
I use XRAY server and would like to compute Test Failure rate for each of my tests automatically and on regular basis.
Test failure rate = (No. of times test failed) / (Total no. of times test got executed)
I can get a similar statistics from Squish Test Center. Please see image below:
I have already tried following:
1. I figured out that I can compute Rate=A/B manually for a specific test (e.g. DEMO-9) using the following JQL
A = [issuetype = 'Test Execution' and issue in testTestExecutions('DEMO-9', 'FAIL')]
B = [issuetype = 'Test Execution' and (issue in testTestExecutions('DEMO-9', 'FAIL') OR issue in testTestExecutions('DEMO-9', 'PASS')]
For this, I referred documentation at the link below:
https://docs.getxray.app/display/XRAY/Enhanced+querying+with+JQL
But there is no way that I could find to do this automatically for all of my tests.
2. I tried but could not accomplish this with Rich Filters on my Dashboard to show list of my tests with a column for "Failure rate".
3. I was not able to find any column on the default XRay Reports or Test panel to get such a statistics. A similar statistics is available on Test Execution Report for "Success Rate" (please see image below) but that is applicable for the entire execution which may contain several tests. I would like to see it for individual tests.
Any help is greatly appreciated. Thanks!
Hi @Vrajesh Dave - welcome to the community
Do you organize your test cases/test phases with TestPlans? This would give you the possibility to automativally get metrics for testcases in your test plan:
https://docs.getxray.app/display/XRAYCLOUD/Test+Plans+Metrics+Report
Please also see the following link to get an overview of good practices and ways to organize your tests in XRAY:
https://docs.getxray.app/display/XRAY/Terms+and+Concepts
https://docs.getxray.app/display/XRAY/Organizing+Tests
https://docs.getxray.app/display/XRAY/Planning+Tests
Hope this was helpful.
Best
Stefan
Hi @Stefan Salzl, Thank you for the warm welcome! I'm excited to be a part of this community.
Yes, we do organize our test cases with Test Plans. I am aware of the general concepts and we do follow most of the good practices. Thanks for the links!
I was not able to get an answer to my specific question from the links though.
Thank you so much for your reply!
Best Regards,
Vrajesh
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
one of the links leads to the Tesplan Metrics which provide Test success rate in the report.
As you organize your test cases in testplans this should be possible with the mentioned link.
In case I got something wrong with your requirement please appologize. May I kindly ask then for further specification in order to try to find an applicable solution.
Best
Stefan
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @Stefan Salzl , thanks for your reply!
I believe you are talking about the documentation as shown in the image below:
I can see this metric in our Test Plan as below:
Unfortunately, this is not the number I am looking for. This Success rate is for the entire test plan and I am guessing it is = (Number of Passed Test) / (Total number of tests in the test plan that got executed)
The number that I am looking for is individual Test failure rate as described in my original query: Test failure rate = (No. of times a particular test failed) / (Total no. of times that particular test got executed). Here this particular test could have been executed in any number of test plans and considers the entire history of its execution.
I hope that this gives a clear picture of my requirement. Thanks for your time!
Best Regards,
Vrajesh
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello @Vrajesh Dave
Did you ever figure out the answer to this one? I have been looking for something similar, but does not seem to be possible
Best Regards
Mathias
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello @Mathias Kristensen , I got to know that this improvement is reported to Xray team via: XRAYCLOUD-4229 and XRAYCLOUD-1662 You may choose to Vote so that they can keep track of our interest and Watch to receive news on the improvement.
Looks like the above improvements are not exactly the same as required. I will post update here in case I find something related. Thanks!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Updates: I checked with Tech support from Xray. They have logged an enhancement request: XRAYCLOUD-7175
You may choose to Vote so that they can keep track of our interest and Watch to receive news on the improvement.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thanks!
I found a way to do it manually by exporting the numbers with xray document generator, and then compute it. But that is a bit cumbersome.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.