Hi, I would like to report both at the test case level and at the test step level. Reporting at the test case level is simple because I can use the comfortalbe Test Runs Reports function. However, I require a more detailed reporting. I.e. reporting on the test step level. I would like to see: * How many test steps in total in a test execution * Distribution per test status of all test steps in a test execution * Possibly other details How do I accomplish this detailed level of reporting with Xray? I'm using Xray 4.0.0 on a Jira 8.5.1 data center edition. Thanks, Sebastian

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Reporting at test step level in Xray

Sebastian Held

Hi,

I would like to report both at the test case level and at the test step level.

Reporting at the test case level is simple because I can use the comfortalbe Test Runs Reports function.

However, I require a more detailed reporting. I.e. reporting on the test step level. I would like to see:

How many test steps in total in a test execution
Distribution per test status of all test steps in a test execution
Possibly other details

How do I accomplish this detailed level of reporting with Xray?

I'm using Xray 4.0.0 on a Jira 8.5.1 data center edition.

Thanks,

Sebastian

Accepted answers

Sergio Freire - Xblend

Hi @Sebastian Held ,

for the time being, Xray does not provide aggregated reporting at test step level.

You can however have some information at the Test Execution issue screen level rightaway. You may include a column named "Steps Progress" that will show a completion percentage as mentioned in the docs:

https://confluence.xpand-it.com/display/XRAY/Test+Runs#TestRuns-ConfigureTestRunColumns

I invite you to reach out Xray support with some concrete examples so the team can create suggestions in our backlog.

Regards,

Sergio

All comments

Sergio Freire - Xblend

Hi @Sebastian Held ,

for the time being, Xray does not provide aggregated reporting at test step level.

https://confluence.xpand-it.com/display/XRAY/Test+Runs#TestRuns-ConfigureTestRunColumns

I invite you to reach out Xray support with some concrete examples so the team can create suggestions in our backlog.

Regards,

Sergio

Sergio Freire - Xblend

By the way @Sebastian Held , as means to better understand and help you out, I would love if you could explain the need behind the step level reports. What are you trying to measure at the end?

What is the need that you’re trying to overcome with those reports?

regards

Sérgio

Sebastian Held

@Sergio Freire - Xblend

Hi Sérgio,

thanks for the response.

I was able to handle the requirement for the time being as I created a script which accesses both the Jira and the Xray REST API in order to retrieve the data which I would like to process. The script was no big deal. However, it would be more convenient to retrieve the data on a test step level using the UI layer.

As for my use case: in my organization we use large sets of end-to-end tests. Therefore, some of the tests have 50 steps or more. The default Xray logic marks a test case as "failed" if one of its test steps falls to "failed". I do not want to change this default setting because it makes sense in general.

Since our end-to-end scenarios are rather complicated, chances are good that the reporting would be misleading, if we only reported at a test case level. Furthermore, since we are in the project phase at the moment, we except about 30 percent of the test steps of a test case to fail. Therefore, if we reported merely at a test case level, the success rate of the testing activities would be zero or close to zero. However, zero is inaccurate as about 70 percent of the test steps fall to "passed".

Thanks,

Sebastian

Sebastian Held

Hi Sérgio,

I just saw that Xray 4.1.0 was released a couple of days ago. Will it be possible to use the document generator to report at a test step level? If so, I could mothball the script accessing the Jira/Xray API right away. Obviously it makes sense to create a reporting template withe the document generator rather than developing and maintaining a custom script.

Thanks,

Sebastian

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

server

xray

jira

jira-server

data-center