Is it possible to configure Xray Overall Execution Status priority (FAIL > EXECUTING > PASS)?

[Siva Kumar]

Hello,

We are evaluating whether Xray Data Center supports a configurable priority model for Overall Execution Status.

Desired behavior:

FAIL > EXECUTING > PASS

Example:

Test A contains multiple Test Executions:

Execution 1 = FAIL

Execution 2 = EXECUTING

Execution 3 = PASS

Expected Overall Status = FAIL

because at least one execution has failed and FAIL should have the highest priority.

Observed behavior:

FAIL + PASS → Overall = PASS

FAIL + EXECUTING → Overall = EXECUTING

This suggests that Xray determines the Overall Execution Status based on the latest/relevant Test Run rather than using a configurable priority model.

During our investigation, we reviewed:

- Set Final Test Run Status

- Inline Execute Tests

- Project-level Xray settings

We also found references to:

"Final statuses have precedence over non-final statuses"

However, we could not find any configuration that allows defining priority among statuses such as FAIL, EXECUTING, and PASS.

Questions:

1. Does Xray Data Center provide any native configuration to define status priority such as FAIL > EXECUTING > PASS?

2. Is there any administrative setting that allows FAIL-dominant aggregation of Test Run results?

3. Can "Final statuses have precedence over non-final statuses" be used to achieve this requirement, or does it only distinguish between final and non-final statuses?

4. If this is not currently supported, is it considered a product limitation, and is there an existing enhancement request for this feature?

Thank you.

[Marc -Devoteam-]

Hi @Siva Kumar 

In addition to @Viswanathan Ramachandran , this is fully true

"Native Status Priority? No. Xray’s evaluation engine is recency-driven (based on the latest Test Run timestamp), not severity-driven.

FAIL-Dominant Aggregation Setting? No. FAIL-dominant logic applies to Requirement Coverage (across different tests), but not across multiple runs of the same test."

This would also be wrong, as multiple test execution overall status could wrong based on status, the last execution should be leading if this is PASS, this should be taken, what if a previous execution coverage from the Requirement shows FAIL, but is 3 weeks OLD, would the Overall Execution status then be FAIL?

[Tomislav Tobijas]

Hi @Siva Kumar ,

As far as I’m aware, Xray Data Center does not provide a native configuration for assigning an arbitrary priority to Test Run statuses, such as FAIL > EXECUTING > PASS.

Although I have not played that much with Xray to be certain. What I'd recommend is For confirmation of whether there is contacting Xray Support, as Xray is a Marketplace app developed and supported by Xblend vendor. 
They will most likely be able to confirm current functionality here and say what's possible and what's not (together with some potential workarounds) 👀

Cheers,
Tobi

[Viswanathan Ramachandran]

Hi @Siva Kumar 

No, Xray Data Center does not provide a native configuration to define status priority (such as FAIL > EXECUTING > PASS) for overall test execution status.

Quick Answers to Your Questions:

1.Native Status Priority? No. Xray’s evaluation engine is recency-driven (based on the latest Test Run timestamp), not severity-driven.

2.FAIL-Dominant Aggregation Setting? No. FAIL-dominant logic applies to Requirement Coverage (across different tests), but not across multiple runs of the same test.

3.Does "Final Status Precedence" Help? Only partially. It ignores non-final statuses (EXECUTING/TODO) in favor of final ones (FAIL/PASS). However, between FAIL and PASS (both final), it will still select the latest run, resulting in PASS.

4.Product Limitation & Workarounds? Yes, this is a limitation by design (Xray assumes the latest run reflects the current state of the app).

⚬ Workaround 1: Instead of creating a new Test Execution for a re-test, update the existing Test Run instance or scope runs by Version/Test Plan.

⚬ Workaround 2: Use ScriptRunner to build a custom field that calculates status based on your priority logic.

⚬ Workaround 3: Aggregate custom status priorities in eazyBI for reporting purposes.