Is there a way compare two Test Executions (xray)?

Question

Hello,

we would like to compare different xray Test Executions e.g. the current one for a dev branch and one for some PR to see how that PR affects the results.

Of course, it would be possible to get this result using the API and do some magic.

But is there a better way to do this, e.g. a plugin or some Jira functionality?

Walter White_n2pumm · Answer

There is a Python script bound to a commit or push hook. I am reading the test results using the Xray REST API for the associated test run for this PR and for the main branch. Then I compare the existence and the results for the tests. Afterwards I put together a Markdown table with the results and add that to the PR as a comment using the Bitbucket API.

pat.mayer · Answer

Hi can you explain how you got it done?

Walter White_n2pumm · Answer

I also asked the xray support and there was not suitable answer.

Our developers must be able to see the impact of each PR on a defined test plan. The baseline is the result of the test plan for e.g. the current development branch.

It's a rather simple thing and I put it together myself using the xray API in a few hours.

The following is now added as a PR comment (with some markdown) once the webhook-triggered tests are done.