Reporting at test step level in Xray

[Sebastian Held]

Hi,

I would like to report both at the test case level and at the test step level.

Reporting at the test case level is simple because I can use the comfortalbe Test Runs Reports function.

However, I require a more detailed reporting. I.e. reporting on the test step level. I would like to see:

• How many test steps in total in a test execution
• Distribution per test status of all test steps in a test execution
• Possibly other details

How do I accomplish this detailed level of reporting with Xray?

I'm using Xray 4.0.0 on a Jira 8.5.1 data center edition.

Thanks,

Sebastian

[Sergio Freire - Xblend · Accepted solution]

Hi @Sebastian Held ,

for the time being, Xray does not provide aggregated reporting at test step level.

You can however have some information at the Test Execution issue screen level rightaway. You may include a column named "Steps Progress" that will show a completion percentage as mentioned in the docs:

https://confluence.xpand-it.com/display/XRAY/Test+Runs#TestRuns-ConfigureTestRunColumns

I invite you to reach out Xray support with some concrete examples so the team can create suggestions in our backlog.

Regards,

Sergio

[Sergio Freire - Xblend]

By the way @Sebastian Held , as means to better understand and help you out, I would love if you could explain the need behind the step level reports. What are you trying to measure at the end?

What is the need that you’re trying to overcome with those reports?

regards

Sérgio 

[Sebastian Held]

@Sergio Freire - Xblend 

Hi Sérgio,

thanks for the response.

I was able to handle the requirement for the time being as I created a script which accesses both the Jira and the Xray REST API in order to retrieve the data which I would like to process. The script was no big deal. However, it would be more convenient to retrieve the data on a test step level using the UI layer.

As for my use case: in my organization we use large sets of end-to-end tests. Therefore, some of the tests have 50 steps or more. The default Xray logic marks a test case as "failed" if one of its test steps falls to "failed". I do not want to change this default setting because it makes sense in general.

Since our end-to-end scenarios are rather complicated, chances are good that the reporting would be misleading, if we only reported at a test case level. Furthermore, since we are in the project phase at the moment, we except about 30 percent of the test steps of a test case to fail. Therefore, if we reported merely at a test case level, the success rate of the testing activities would be zero or close to zero. However, zero is inaccurate as about 70 percent of the test steps fall to "passed".

Thanks,

Sebastian

[Sebastian Held]

Hi Sérgio,

I just saw that Xray 4.1.0 was released a couple of days ago. Will it be possible to use the document generator to report at a test step level? If so, I could mothball the script accessing the Jira/Xray API right away. Obviously it makes sense to create a reporting template withe the document generator rather than developing and maintaining a custom script.

Thanks,

Sebastian

[Sergio Freire - Xblend]

Hi  @Sebastian Held 

what concerns the e2e scenarios with all those steps, my recommendation is to make simpler test scenarios and test features in isolation with specific test cases and for the e2e test the steps can be more high-level as you would test individual things in specific test cases. 

Concerning Xray 4.1.0, it comes with the built-in document generator. The capabilities it provides, in terms of iterating over the data, are the same as you have for Xporter (a standalone plugin). With it, you should be able to iterate over test step and do some reporting with it using the scripting capabilities provide by document generator. 

I would recommend you to reach out Xray support, detailing exactly the report and the dimensions you need and the team will be glad to help you out.

Regards

Sergio

=======================================
Personal blog on testing, Agile and software development: https://sergiofreire.com
Follow me on Twitter: @darktelecom

[akshay kumar]

What is this, i am expecting some clue and here i am getting promotion