Testing - How to run multiple test executions for same x-ray test

[Tom Wood]

I have multiple tester based around the UK, testing at different times. I have set x-ray tests and created multiple test executions per test. My thought being that each tester would be able to complete a test execution in the hope that only once all the test executions have been set to "Pass" would the x-ray test then change to pass

[Stefan Salzl · Accepted solution]

From my point of view:

The testcase itself is "just" a description of "which steps needs to be done" to test a special feature/functionality.

The test execution is then an abstraction of the testcase for a special setting (eg. version --> I execute the steps from a testcase with application in version 1). The testcase then reports the "current" status. 

So if the test execution with version 1 passes the testcase would say "green - passed". After deploying version 2 of the application, creating and executing a new test execution for version 2 that fails the test case will say "red - failed" because the current state is that this feature is (obviously) not working with the current version (2) proved by the test execution.

[Tom Wood]

Stefan - thanks this has given me food for thought on I approach this. Many thanks

[Stefan Salzl]

Awesome. Thanks for letting me know and really happy I could give a helpful input :)

[Chris Kemp]

@Stefan Salzl is it correct that a Test Execution can only contain one instance of a (named) test?  Therefore, to execute the same test a second time, you have to create a separate Test Execution?

If so, this solution does not fit the way in which the part of my organisation works.  If I raise a defect, it is likely to be resolved in-house on an ad-hoc basis ('there and then').  In this scenario, I would prefer to re-run the test from the same execution, keeping the prior instance for historical record (of the defect).  This would allow the Test Execution to have an overall status of Passed.

I guess I am stuck having to create multiple Test Executions, each with a separate Status, containing a different number of Tests.

I appreciate that this is not Agile and so Jira doesn't fit the expectation.
I could accept creating an entity such as a Test Plan or Test Set, having multiple Test Executions, utilising the Test Plan/Set to track the current status of all Tests, but it does not appear that Jira/Xray supports this approach either.

<<EDIT>> I have been working through this as I have been editing this response.
I have now found that Test Plan is split into two different views, Tests and Test Executions.  It appears that Tests only shows the latest Status of each test (each test is only listed once in this view, being a cumulative view of all tests within the Test Executions), which is exactly what I need.

I would prefer to execute my tests from a single Test Set, but I guess having multiple Executions is more 'descriptive'.  I will get used to it :)

[oliver.schulz]

@Chris Kemp but it still creates a new 'Test Execution' ticket everytime you retest a Test, correct?

We are thinking about using Xray, but that seems odd.

For my understanding, let's say we have a test plan with 4 test executions (one per team) and each execution has mutiple tests. If a test fails and I want to retest only this one, it will create a new execution so that I have 5 in total.

[Chris Kemp]

I am slightly confused by your use of 'it will create a new execution'.

Someone must create a Test Execution (issue type), in which 'instances' of tests can be added (either individually, or by test set).  A test instance can only be executed once within each Test Execution (or the Actual Results could be updated, losing the previous test results).

To re-test, you need to create a new Test Execution, and add the test once more (creating a new instance of that test).
FYI I have started naming my Test Executions prefixed with ex01, ex02 , and add them to a Test Plan from which I can report on the current status of all the tests.
Actually, I am now adding a shortened date to the name of the execution ex0622-1.  This allows me to quickly see the 'Test Runs' in date order (from within the XRay Test as stored in my repository).

[oliver.schulz]

Thank you @Chris Kemp , your answer helped me a lot.

With 'it will create a new execution' I actually meant 'I have to create a new Test Execution' like you guessed correctly.

So my structure:
- Test Plan Software Version 0.1
And under the Test Plan:
- Test Execution Software Version 0.1- Tests for Team A
- Test Exection Software Version 0.1 - Tests for Team B

is not good for organizing the testing phase, because as soon as I want to re-test a failed tests, I have to create a new Test Execution.

Hope this helps, but you answered my question anyway :)

[suresh.m.mani]

I have created test execution for round1 under test plan.  Lets say total Test case count is 100, in which 50 TCs is passed  and 50 TCs is failed.

I want to execute the failed test cases on the same JIRA key used for test execution round1, but we can't re run it with the same test execution JIRA key.   

so we need to create another execution JIRA for next run.

If run with another JIRA for failed test cases, do we need to add the failed test cases or overall test set.?

If move the failed test case only for execution, how we will get a view of overall test cases status with already passed test cases.

[Chris Kemp]

I think you are asking the same question as answered in my replies to this discussion.  You need to create a Test Plan and add the Executions.  This will give you the overall status.

NB it isn't that clever, if you edit the oldest Test (run), it will be seen as the latest update, and the status will be taken from that.
ie if Test (run) Execution-1 failed, Test (run) Execution-2 passed, Test Plan would correctly show overall status Test (run) as Passed, but if you edited Test (run) Execution-1 after this time (for whatever reason, not changing the result), the Plan would show the overall status of Test (run) as Failed.

If you use something similar to the suggested naming convention, you can see the Test Executions of each Test, and the name will give you a 'quick view' of when you ran it.

BTW You posted an Answer to the initial question.  You need to post a Reply to 'someone' if you are asking a question.

[Jan-Hendrik Rolf]

Hello!

When do I have to use existings test executions and when do I use new test executions?

[Chris Kemp]

I am not a Jira or xray expert, but I believe your question is answered by considering that the tests (consider these a single instance to be executed, ie runs) within the Execution, can only have 'one result'.  This also includes each step only having one Actual Result.  Therefore, if you 're-use' the existing execution, you will overwrite a previous result.

Personally, I much prefer ALM, in which I can simply run the test again, creating another instance (seeing a history of those test runs within the Test Lab).

Anyway, to have distinct runs of a test (ie with its own results), you would need a separate executions.

NB Having all these executions is the reason for the suggested naming convention, as you can quickly select the execution by viewing Test Runs from within the actual xray test (being a selectable 'view' from Test details).  exmmyy - 'description' is fine for me, but you might prefer exddmmyy depending on how may iterations/runs you expect.

Hope that helps.