Jira Xray vs Jira dashboard anomaly

Bence Juhász

I would like to briefly describe an issue I encountered with the Jira Dashboard while trying to present the test execution results from Xray at project level.

When testers execute a project's test set, they create a corresponding Test Execution in Xray. In this particular case, the focus was mainly on mobile testing, so the tester created separate Android and iOS executions.

Multiple versions of the Android and iOS executions may exist because, when a larger batch of bug fixes is delivered, the tester creates a new execution to retest the affected functionality. This approach ensures that the results of previous test runs are preserved.

Since reviewing multiple executions in Xray can be difficult for stakeholders without detailed testing knowledge, I wanted to use the Overall Test Results gadget on the Jira Dashboard to provide a clear overview of the project's testing status.

However, even when configuring the dashboard to display the latest execution result for each test case, the gadget produces results that do not appear to be logically consistent. The status calculations seem to be incorrect, resulting in misleading and inconsistent test reports being presented to the project team.

The inconsistency is particularly concerning because the displayed results have changed multiple times within a few minutes simply by refreshing the dashboard, even when no testing activity was taking place.

I have discussed the issue with my colleagues, reviewed Xray tutorial videos, and consulted AI, but I have not yet found a solution that would allow us to report accurate test results without introducing significant additional administrative overhead.

Potential workarounds include duplicating test cases or assigning labels to individual test cases. However, both approaches would create a substantial amount of additional maintenance for every project.

Expected outcome: The Overall Test Results gadget should accurately and consistently reflect the latest execution status of each test case, without requiring significant additional manual administration.

Daniel Brooks

This pattern usually comes from the scope of the gadget rather than the test cases themselves, so it is worth narrowing it down before duplicating anything.

A quick isolation test: point the Overall Test Results gadget at a single Test Execution only, then refresh the dashboard a few times over ten minutes. If the numbers stay stable, your executions are fine and the problem is that the wider scope is pooling several Android and iOS executions together. If a single execution still changes between refreshes, note the times and screenshots, because that points to Xray recalculating in the background and is one for Xray support.

Two things that made our reporting stable:

1. "Latest" is worked out from the test runs in the executions in scope, not from which execution issue you created last. A retest execution whose runs are still open, or whose runs were started earlier but updated later, can win or lose against the original execution. Finish the runs in superseded executions and move those executions to a final status before you report.

2. Report from one Test Plan per release instead of from the whole project or a broad filter. Add only the current Android and iOS executions to that Test Plan and base the gadget on the plan. The older retest executions stay preserved for the testers, but they stop competing in the stakeholder view. If you prefer a filter, add one label to just the two current executions and filter on that — labelling two executions is much cheaper than labelling every test case.

Also check whether Android and iOS are set up as test environments in Xray. If they are, the same test can legitimately count once per environment, so the gadget total can be larger than your number of test cases. Decide whether stakeholders need a per-environment view or a combined one, and keep the gadget set up for just that one view.

Hope that helps narrow it down — the single-execution test will tell you which half of this applies.

One last practical note from our side: once the gadget scope is fixed, write the chosen setup down where the whole team can find it — which Test Plan you used, which environments are in scope, and who closes out the old executions — so the dashboard does not drift again next release. That short checklist has saved us re-checking the same numbers every sprint. It is a small habit, but it keeps reporting trustworthy in a small team's workspace.