Test case management/exec with Xray

[Bob Boursaw]

Hello, 

Very new to all this and getting a bit lost in the docs, tutorials, and other info (some very outdated).

I am trying to get a good grasp on managing test cases within Jira using Xray and then executing said tests (many manual, at this time)

I believe I understand the basics of creating the tests, organizing them via the test board and adding tests to plans.  Here is where things break down a bit given what I am seeing compared to docs and and videos I'm finding.

1)  I don't see anywhere in any of my screens where you can assign a start/end date to a Test Execution.  Does this only show up once you've added the test execution to a sprint and then start said sprint?  Do you have to add this field in?  

2)  Do you associate a test execution with a plan?  Or is the execution at a lower level, closer to tests?  i.e. You associate tests and test sets with executions?  

3)  With a scrum project in Jira, do you add things such as the Test Plan and/or Test Execution to your sprint backlog?  Or is that not necessary?  Seems like a dumb question, but would rather know than assume incorrectly

4)  Do you have to manually assign tests/test sets to individual testers?  Historically, our QA team just worked through different sets of tests on their own and they weren't specifically assigned.  Possibly this is just a bad practice and we need to change, or is there a way to leave things a bit more open with Jira/Xray?

If anyone has any links to good beginner/get you going type videos that are applicable to the current versions of the software, that would be awesome to see.

Or, if somebody could throw me a few bullets on what the basic steps are to get rolling with this process, that would be awesome.  I feel as though I'm hitting a bit of analysis paralysis with all of the different approaches, docs, videos, etc., and am not making much headway.

Any assistance would be greatly appreciated!

Thanks all,

Bob

[Rawan Bardini · Accepted solution]

Hi Bob,

There is all sorts of info out there about Xray and usage, possible ways to set things up, etc, as you have mentioned. It can be overwhelming I agree. :) The goal I would say is to find something that is going to work for your organization, and drive the most value. Versus trying to copy something generic. I also would say that analysis paralysis is real, and the best way to identify what is going to work for you or not is to pick a direction, try it out, and then iterate as you need.

This is a good into overview video from Xpand IT on Xray that was posted only a year ago, so it is relatively up to date. Not sure if you have already seen this but it is a good demo of the product overall. Xray Product Walkthrough

If you have not already, I would also recommend looking through the Xray User Guide, specifically in the TTT (Tutorials, Tips, and Tricks) section. (note there is a Server and Cloud version of the user guide so make sure you look at the right one).

To help with some of your specific questions:

1. I am not familiar with a Start/End Date captured on the entire Test Execution. The system will of course capture in the history of the Test Execution when it was moved out of "Open" into a resolved status of some kind. Now within a Test Execution, you have a list of Test Runs. Each of these Test Runs when executed will capture the Start/End Time automatically. You would see that time stamp noted on the "Execution Page" of each Test Run (when you click the play button on the Test Run in the Test Execution, it takes you to the Execution Page). 

2. You can associate many Test Executions to a Test Plan. The basic taxonomy of the issue types in Xray are Tests, which are then put into Test Sets for organization (think of it like a folder structure almost). Those have nothing to do with execution, aka running a Test. When you want to execute Tests, then you would use Test Executions (sub test executions are just a sub-task of a Story, otherwise the functionality is the same). That will let you assign status values to your Test Runs (like pass, fail, etc). You can add Tests individually 1 by 1 to a Test Execution, or you can add an entire Test Set (folder of tests) to the Test Execution, which of course would be easier. Now if you like you can also use Test Plans to aggregate your Test Executions into 1 place, so you have 1 view of "testing status" if you will across all of your Test Executions. 

3. Depending on how you set up and handle your scrum projects, there are many ways to incorporate the Xray issue types into your sprints. Technically at a Story level you could use Sub-Test Executions to execute the Tests linked to a Story, and that again is a sub-task of your Story. So would be tracked like any other sub-task in your Story on the board. If you have testing outside of a Story, then you would use the Test Execution issue type to represent your testing work in the sprint and ideally yes see that on the board to capture the testing effort for the team (and the tests they are executing). Usually a Test Plan would not be on the board, but more to consolidate the Test Executions ran during the sprint for example into 1 place. Many ways to manage Xray and the types, so again depends on how you structure things for your organization. You can also limit or suppress what appears on a board in Jira (via JQL query), so can be something that every scrum team may want to handle differently based on your organization. 

4. In terms of Assigning Tests, that is as you said very flexible in many ways. You can assign a Test, you can also assign a Test Execution, and then you can even assign each individual Test Run within a Test Execution. :) Many ways to work with an Assignee value, but really depends on what you are trying to capture to determine which level of assignment you want to do. Also know that Xray will automatically capture Executed By on each Test Run depending on who updates the Execution Page. You can then also show this field in the Test Execution, where the Test Runs are listed, by adding in another column to that list.

Hope that helps you somewhat! I would again suggest just picking something, trying it out, and then iterating. Good Luck! :)

[Bob Boursaw]

Thank you!  I very much appreciate your time and answers to my questions.  I will continue to work through the guide and check out the video link.  

[Bob Boursaw]

Quick follow-up, if I may.

Just curious if I am on the right path here / if this makes sense.

Goal:  Ensure new feature tests are covered and all other tests (regression) are completed within the defined sprint.

So I believe I would first create a test plan for the version of the software under test.

Next I would create 2 test executions

Test exec #1 for all tests in my test set called 'Regression'

Test exec #2 for all tests in my test set called "Feature"

During the process where I create each of the test executions, I would associate them with the test plan mentioned above.

So now, I have my test plan with its associated test executions that reference all of the tests that need to be tested.  

During my sprint planning process, wouldn't I need to include one of the test entities in my sprint backlog? (Test Plan, Test Executions, etc. ?) I am missing how the testing progress is going to be tracked during the sprint/as one of the sprint tasks, if that's not done in some fashion.  Or, would I do something like creating a user story for the testing that needs to be done, set a requirement for that story to all tests, and include the story in my sprint?

Thanks!

Bob

[Rawan Bardini]

Hi Again Bob,

Overall what you have described here is definitely valid and you are absolutely on the right path. You can do exactly as you have listed here, or you could even create the Test Execs first and then the Test Plan, either way you have the same taxonomy/structure.

Now when it comes to Sprint planning and the process for showing work on the backlog, you would be using the Test Exec to represent that work yes. Although how the work is tracked and represented will really depend on your process within your scrum team.

Good practice in Agile is to have a Story on the board represent all the work that needs to be completed before the story is "done", which should include testing (maybe different at your organization). If you take that approach, then all your new feature tests should really be associated to each individual story, and then Sub-Test Execs would be created for each individual story. That way Sub-Test Execs are treated as a sub-task of the Story, and those feature tests are basically run as part of the story already as it shows on the board. If you have swim lanes on the board you can also specify by sub-task and see the Sub-Test Exec on the board as part of a story. 

It is usually discourage to use a Story to represent testing as that is not technically a change we are making to the system, which is why ideally each story is being tested by a Sub-Test Exec (testing is part of the Story being considered done).

Now for Regression, to your point, there are no Stories, so in that case yes your Test Exec would be assigned a Sprint value and reflect on your board just like any other issue in the Sprint. Same fields are available like Points, Fix Version, etc, and could be treated basically like a Story but capturing the Regression test effort for the Sprint.

Of course depending on the Agile guidance within your organization you may have many approaches to how testing is tracked as part of a team's sprint.

Hope that helps!

[Bob Boursaw]

That definitely helps, thank you.  I'm still a little shaky on exactly what a sub test exec is and how it differs from a test execution, but will read up on that further.  Is it simply the context in which a test execution is applied?

[Rawan Bardini]

Yes that is it exactly! :) You got it!

A Sub-Test Exec looks and behaves the same exact was as a Test Exec. The only difference is that Sub represents a Sub-Task of a Story, so that it literally has to be completed before the story can progress to being "done" (just like any other sub-task on a Story).

The Sub-Test Exec would also only be focused on the tests related to that one story or that one feature. So a very limited scope of tests to execute. 

Otherwise the functionality from a tooling perspective is the exact same.