Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
Community Members
Community Events
Community Groups

Testing - How to run multiple test executions for same x-ray test


I have multiple tester based around the UK, testing at different times. I have set x-ray tests and created multiple test executions per test. My thought being that each tester would be able to complete a test execution in the hope that only once all the test executions have been set to "Pass" would the x-ray test then change to pass

1 answer

1 accepted

0 votes
Answer accepted
Stefan Salzl Community Leader Dec 20, 2021

From my point of view:

The testcase itself is "just" a description of "which steps needs to be done" to test a special feature/functionality.

The test execution is then an abstraction of the testcase for a special setting (eg. version --> I execute the steps from a testcase with application in version 1). The testcase then reports the "current" status. 

So if the test execution with version 1 passes the testcase would say "green - passed". After deploying version 2 of the application, creating and executing a new test execution for version 2 that fails the test case will say "red - failed" because the current state is that this feature is (obviously) not working with the current version (2) proved by the test execution.

Stefan - thanks this has given me food for thought on I approach this. Many thanks

Like Stefan Salzl likes this
Stefan Salzl Community Leader Dec 23, 2021

Awesome. Thanks for letting me know and really happy I could give a helpful input :)

@Stefan Salzl is it correct that a Test Execution can only contain one instance of a (named) test?  Therefore, to execute the same test a second time, you have to create a separate Test Execution?

If so, this solution does not fit the way in which the part of my organisation works.  If I raise a defect, it is likely to be resolved in-house on an ad-hoc basis ('there and then').  In this scenario, I would prefer to re-run the test from the same execution, keeping the prior instance for historical record (of the defect).  This would allow the Test Execution to have an overall status of Passed.

I guess I am stuck having to create multiple Test Executions, each with a separate Status, containing a different number of Tests.

I appreciate that this is not Agile and so Jira doesn't fit the expectation.
I could accept creating an entity such as a Test Plan or Test Set, having multiple Test Executions, utilising the Test Plan/Set to track the current status of all Tests, but it does not appear that Jira/Xray supports this approach either.

<<EDIT>> I have been working through this as I have been editing this response.
I have now found that Test Plan is split into two different views, Tests and Test Executions.  It appears that Tests only shows the latest Status of each test (each test is only listed once in this view, being a cumulative view of all tests within the Test Executions), which is exactly what I need.

I would prefer to execute my tests from a single Test Set, but I guess having multiple Executions is more 'descriptive'.  I will get used to it :)

@Chris Kemp but it still creates a new 'Test Execution' ticket everytime you retest a Test, correct?

We are thinking about using Xray, but that seems odd.

For my understanding, let's say we have a test plan with 4 test executions (one per team) and each execution has mutiple tests. If a test fails and I want to retest only this one, it will create a new execution so that I have 5 in total.

I am slightly confused by your use of 'it will create a new execution'.

Someone must create a Test Execution (issue type), in which 'instances' of tests can be added (either individually, or by test set).  A test instance can only be executed once within each Test Execution (or the Actual Results could be updated, losing the previous test results).

To re-test, you need to create a new Test Execution, and add the test once more (creating a new instance of that test).
FYI I have started naming my Test Executions prefixed with ex01, ex02 , and add them to a Test Plan from which I can report on the current status of all the tests.
Actually, I am now adding a shortened date to the name of the execution ex0622-1.  This allows me to quickly see the 'Test Runs' in date order (from within the XRay Test as stored in my repository).

Like # people like this

Thank you @Chris Kemp , your answer helped me a lot.

With 'it will create a new execution' I actually meant 'I have to create a new Test Execution' like you guessed correctly.

So my structure:
- Test Plan Software Version 0.1
And under the Test Plan:
- Test Execution Software Version 0.1- Tests for Team A
- Test Exection Software Version 0.1 - Tests for Team B

is not good for organizing the testing phase, because as soon as I want to re-test a failed tests, I have to create a new Test Execution.

Hope this helps, but you answered my question anyway :)

Suggest an answer

Log in or Sign up to answer