Xray Issues Workflow

[Cromie]

We just installed Xray and I have a few questions about the workflow and how everything ties together. I understand adding -> Test(s) -> Test Set -> Test Plan -> Test execution (or various combinations of these) for the first version release, but what about when you want to re-use a test plan or test suits for the next release, how does that all work or can you? 

Example: I created my tests, created a test set added it to a test plan, and then executed this for a version release. It all looks good, I can see my results, etc. Now we have another release and I want to run the same tests. If I add the test plan or create a Sub Execution of the test sets to the new release story/or task, it indicates I have no Tests associated with it. Is this something I should not be doing? Do I need to start all over again by adding Test(s)? 

[Rawan Bardini · Accepted solution]

Hi Cromie. As you know Xray can be customized and used in many different ways depending on your organizational needs and processes. I cannot necessarily answer in regards to your exact needs, but will try to do so in general and based on your example.

In general, a Test should be created, then added to a Test Set. You can think of a Test Set almost like a folder structure to store and organize your Tests. Similar to the Test Repository, the difference being that a Test can belong to 1 or many Test Sets. You can then also directly link Tests to Stories or whatever you are using for requirements and then use those to indicate Test Coverage of the requirement. Now when it comes to executing or running your Tests (pass/fail, etc) that is when you would use either a Test Execution or a Sub-Test Execution. The only difference being a Sub-Test Execution is a sub-task of a Story or Task (or whatever you define for your organization), meaning the Story cannot be marked "Done" or "Complete" in Jira until the sub-tasks are completed, making the Sub-Test Execution a sub-task of the Story. A Test Execution is more so used for testing efforts outside of a Story (in our company we use it for standalone testing like Regression, E2E, UAT, those efforts outside of the scope of a Story for example). Now a Test Plan is what you would use when you have many Test Executions (or sub-test executions) that you want to combine together in order to get 1 cumulative/consolidated test status/view.

Given this general information, in your example you mention creating tests and adding them to a test set, and then you are adding them to a Test Plan directly. I would question if you should be using a Test Execution instead of a Test Plan at that point to run/execute your tests. Then when/if you have multiple Test Executions in your release you want to consolidate together, then add those Test Executions into a single Test Plan. 

As you said you can also create a Test Execution/Sub-Test Execution from a Test Set. This is the easiest way to group together and execute the tests you would like to for any given release. You can then clone/copy the Test Execution if you like in the future to run the same exact tests, or keep updating your Test Set and then in the future create a Test Execution directly from the updated Test Set. Doing it this way you are not having to start all over again in terms of identifying the tests you would like to execute. 

When it comes to Test Coverage though, you will need to link the Tests to the Story/requirement in order to see the Test Coverage for that specific requirement. This is something that you would do for every requirement, and the assumption here makes sense when you are taking an Agile approach to software development as each Story/requirement can have different tests to execute. Now if you are taking more of a waterfall approach, I guess it would be possible to clone the Story/Requirement, and keep all the linked Tests that way instead of having to link them manually every time. 

Others may also have comments/observations for you, but as I said I am trying to answer here in a general way and give some general guidance based on your question and specific example. Hope this helps. You can also reach out to the Xray Product Support team directly if you have more detailed questions or need more guidance. Thanks!

[Cromie]

Thanks Rawan, all this is very helpful. To your one point above about clone/copy of test executions, I never thought about that. I will test this all out.

Question, do you close your test sets, test plans once you created them or do they stay open? 

Thanks again! 

[Rawan Bardini]

Glad I could help!

In our organization we do not close out Test Sets. We use them basically as a folder structure/repository to organize our Tests. We have a Regression Test Set as well per system that folks can then use to maintain and easily execute Regression Tests. We do have a Deprecated status for our Tests that we use to indicate when a Test is no longer valid or executable for any reason. This is what we then use to remove Tests from scope of our metrics, and to keep Test Sets up to date (we remove Deprecated Tests from the Test Sets).

Test Plans are something we do have a Done status for, to indicate when we are complete with the Test Plan (when all the Test Executions within it are also considered Done).

Depending on if you have the Cloud or Server version of Xray, there is also some good guidance and documentation in their User Guide (this is the link to the Cloud one), which presents some of the options around Workflows and other ways to manage the Xray Issue Types within your Jira Projects.

Good luck testing things out!

[Cromie]

Thanks Rawan, your help is much appreciated. 

[Cromie]

Hi Rawan, for the tests and test sets that are created, do you just leave them unassigned in your backlog?  

[Rawan Bardini]

Yes, we do not typically assign the Tests or Test Sets. We do assign the Test Runs within the Test Executions typically, or even the Test Executions themselves depending on the use case.

In terms of the backlog, we leave it up to each scrum team if they want to see their Tests and Test Sets in the backlog at all. Most of our teams update their backlog filter to remove the Tests and Test Sets, and only keep the Test Executions in the backlog. Although every team is different and has a different preference. 

[Cromie]

Makes sense. Its good to know what works well for others, thanks for sharing. 

[Cromie]

Hi Rawan, in testing things out I noticed if I need to execute the same test in the same test environment (say its UAT for the next release), when I link the Test to my new Task it's already marked Passed because that test was already run in that environment the previous release. Did you run into this situation when testing things out?  

[Rawan Bardini]

Yes that is true. I assume you are looking in the Test Coverage section? If yes, then you will notice under the Analysis & Scope section that it most likely has "Latest" selected. So that means it will show the Latest execution status or run status of the Test. There should be an option below that for Test Environment as well, which is most likely set to All Environments. When you configure your environments for a Jira project you should be able to then select the one you want to see the result for here. That should address your issue, so you would see the Latest result, but if you selected the UAT environment let's say, then it would show only the result for that environment.

Of course there are other ways to use this section, using Version or Test Plan which could be another way to analyze your results. All depends on what you are trying to see. 

Hope that helps.

[Cromie]

Yes, the Test Coverage section. I see... once I add the versions I will select the version and then the Test Environment. Then it will be different every time based on the version. I really appreciate the help. I've watched most (if not all) the video's, but they mostly just scratch the surface of the tool. Thanks again. 

[Cromie]

Hey Rawan, quick question.  I added a few new Test Environments, however, I can't see them in my Associated Test Environment (Test Execution) list. They show in the Test Environment list under Test Coverage as expected. Note, I added some a few weeks ago and they were showing as expected in both lists. Have you run into this issue before? 

[Cromie]

I think I just figured it out. The environments were created but not added to the project. Once I added them, they showed in the Associated Test Environment list for Test Executions.  

[Rawan Bardini]

Yup, you got it! :)

[Deepa Nair]

hi Rawan,

Thanks for your inputs. I had some questions around a Test Plan mapped with multiple Test Execution Cycle - does the Test plan board status displays the latest test execution cycle results or the consolidated results of both the the execution cycles? Is there a way to track the individual Test Execution Cycle status on the Test Plan board view along with the folder structure?

[Deepa Nair]

I have one more question - every time we create a Test Plan if we need to maintain the folder structure from the repository but not all the test cases from there ( say I need to create a Test Plan for P1 only TCs and then another Test Plan for P2 only test Cases) what is the most optimal way to do that?