Zephyr reporting by sprints

Hi,

We'd like to use the Zephyr add-on together with Jira-Agile. While Zephyr enables us to create the test cases, I'm having troubles with the Test Summary and Reporting side in our Agile environment. While I can link Test Cases to Stories, the summary/reporting by sprint seems quite difficult.

How can I create a report which shows how many Test Cases are covering each user story and what their current/latest execution status is? This will help me determine which story is not covered by tests yet or needs more testing effort. Can I filter this by sprint?

How can I creat a report/summary which shows which/how many Test Cases are in the current sprint, based on the stories in the current sprint? This will help me oversee the work by sprint.

Given that we have >100s Test Cases, how can I easly create a Test Cycle based on the stories of the current iteration? Can I filter the Test Cases by sprint?

Can I create Test Cycles by Sprint and another parameter (e.g. component, team)? Having only one level of grouping makes it very difficult to create meaningful cycles as we have multiple teams working in every sprint. Ideally I would like to have a Test Cycle by Team for each Sprint, whereby the Sprint is the parameter from Jira-Agile. I find Version in an Agile environment not very useful.

How can I create a report to show which of the Test Cases are currently not linked to a story or requirement? This willl help me catch any tests which are not currently linked.

I know these are a lot of questions, however I was hoping that Zephyr will help me manage these aspects and at the moment I don't seem to be able to do any of these.

Thanks for any help.

Cheers,

Tom

5 answers

Hello Thomas...

Your questions are many and I have lengthy process configuration answers/discussion for you (and others). My response will generally be in the direction of addressing these points in your post...

  • 'the summary/reporting by sprint seems quite difficult'
  • 'How can I creat a report/summary which shows which/how many Test Cases are in the current sprint, based on the stories in the current sprint?'
  • '...help me oversee the work by sprint'

Please bear with me if some of my response comes across a bit unclear. My coworkers have collaboratively fought the 'process and configuration legion' repeatedly, learning together and honing how we conduct work. It's an ever-growing experience.

The information this post contains would really be more suited for a live demo or process series (which I'm open to write). I'll can explain more if you (and others) have questions.

We run tight sprints in my workplace. Development team progress is tracked through JIRA Agile. 'Zephyr for JIRA' (Z4J) is used for manual test cases and cycles. Confluence, and other various Atlassian tools/plug-ins/add-ons are used for reporting, documentation, and customer interfacing.

As a QA/testing manager I need the ability to track all development sprint activity, monitor all testing being executed on the development items in each sprint (and other testing coverage), observe overall QA/testing team activity, provide reporting details for a number of metrics... and more.

I'll try to describe how I've got things set-up to provide visibility across both development and testing realms using JIRA Agile and Z4J. I'll focus this post on how JIRA Agile relates to development and testing groups where I work, and break it down in sections...

JIRA AGILE - Development VS. QA/Testing Board Configurations:

First, as mentioned, JIRA Agile is used by everyone in my shop... from development groups to testers. We all (devs/testers) have worked together to create numerous filters. Some of those filters help developers only see development issues in their Agile projects. This helps each project filter out Z4J test cases from their backlog and JIRA Agile views... then use one or more Agile board/s to capture their development items during sprints.

So, as a QA/test manager I ALSO use a custom JIRA Agile board I've configured for my testing team (ACTUALLY one board for Scrum, and one for Kanban, but that's a whole other discussion). My team's Agile board still ONLY contains development issues, BUT is configured in a way that makes viewing the issues relevant for QA/testers...

JIRA AGILE - QA/Testing Board SWIMLANES:

I'll start with 'Swimlanes'. I use multiple swimlanes in the QA Agile board. Each custom swimlane is for each project (or a group of collective sub-projects). All of my swimlanes use concise, custom filters I created to capture those development items for each project (or, again, a group of collective sub-projects). I DO NOT TRACK Z4J TEST CASES using JIRA Agile (there are numerous reasons for this, bigger discussion, maybe another time). I track Z4J test cases separately, by different means.

Creating multiple swimlanes enables my one-stop QA/Testing JIRA Agile board to have multiple collapsible/expandable sections. Those readily identified sections/swimlanes contain NOT ONLY those development tickets/issues respective to each unique project (or development group), BUT ALSO those development issues being worked in the active sprints (next section). This 'swimlane' setup makes EVERYTHING centrally located and rapidly viewable.

JIRA AGILE - QA/Testing Board ACTIVE SPRINTS:

When each project starts a sprint that active sprint name appears at the top of my QA Agile board. When I select that sprint my board efficiently filters down to JUST those development tickets related to that project (or that group's active sprint). This also leads to the same reason why the swimlanes and filters become important... for rapid, one-board, all project, quickly identifiable viewing.

JIRA AGILE - QA/Testing Board COLUMNS:

Next up are the 'Columns'. I have five colums set-up that are all based on the status of a development ticket with respect to testing responsibility. The columns are named (left to right) "Pre QA", "Almost QA", "In QA", "Blocked", "Done". The development tickets move across my team's agile board columns depending on the status (workflow) of that issue. The testers know what QA/testing actions they should be taking on a development issue, based on the column where that development issue is sitting.

Here's a process example... when a development issue (ex. 'Story') hits the "Almost QA" column... my team knows the issue has been mostly completed/developed, is being code reviewed, and the very next stage/column will be "In QA". So, to quickly back up, the team begins writing and linking Z4J test cases AS SOON AS the sprint becomes active on day one (when the development issues are visibly in the "Pre QA" column). By the time a development ticket hits the "Almost QA" column, test cases should already be prepared, final test case reviews conducted, steps should be finalized, etc... because that development ticket is coming to QA ownership for testing REALLY soon.

So what happens with the Z4J test issues (cases/executions) that QA/testing has prepared and linked to the development issues?

I can continue covering that in a future post if there's interest.

Cheers.

Hi James, You should write a book on "How to organize work in Agile project using JIRA and Zephyr" topic (smile) So then you have following structure, correct me if I am wrong: 1.Separate Agile Board for Developers with Stories (+Tasks, etc) with its own columns like "Ready for DEV", "IN DEV", "DEV Complete" 2.Separate Agile Board for Testers with the same Stories from Dev Agile Board with its own columns - "Pre QA", "Almost QA", etc 3.Columns "states" from each Board correspond to each other. For example -"Ready for DEV" corresponds to "Almost QA". Column state in one Board may correspond to multiple column states in other. 4.Story`s workflow (statutes) remains the same for both Agile Boards For comparison, here is my approach: 1.One Agile Board for Devs and Testers 2.Agile Board shows Stories and Tasks (no Zephyr Test Cases) 3.Story have following workflow: -Open -In BA review -Acceptance Criteria Definition (Test Cases are written here) -Ready for Dev -In DEV (Sprints starts here) -DEV Complete -Ready for TEST -Testing -PASSED -Closed 4.Agile Board has the same state Columns Can you list pros and cons for each of them and take a comparison also? I am very curious how you manage linkage between Stories and Zephyr Test Cases? And I still don`t get how you link Zephyr Test Cases to a Sprint? Can you recommend some reading on this kind of topics? I certainly lack some basis. Regards.

Hi James,

You should write a book on "How to organize work in Agile project using JIRA and Zephyr" topic smile

So then you have following structure, correct me if I am wrong:

  1. Separate Agile Board for Developers with Stories (+Tasks, etc) with its own columns like "Ready for DEV", "IN DEV", "DEV Complete"
  2. Separate Agile Board for Testers with the same Stories from Dev Agile Board with its own columns - "Pre QA", "Almost QA", etc
  3. Columns "states" from each Board correspond to each other. For example -"Ready for DEV" corresponds to "Almost QA". Column state in one Board may correspond to multiple column states in other. 
  4. Story`s workflow (statutes) remains the same for both Agile Boards

For comparison, here is my approach:

  1. One Agile Board for Devs and Testers
  2. Agile Board shows Stories and Tasks (no Zephyr Test Cases)
  3. Story have following workflow:
    1.  Open
    2.  In BA review
    3.  Acceptance Criteria Definition (Test Cases are written here)
    4.  Ready for Dev
    5.  In DEV (Sprints starts here)
    6.  DEV Complete
    7. Ready for TEST
    8.  Testing
    9. PASSED
    10.  Closed
    11. Agile Board has the same state Columns

Can you list pros and cons for each of them and take a comparison also? 

I am very curious how you manage linkage between Stories and Zephyr Test Cases?

And I still don`t get how you link Zephyr Test Cases to a Sprint?

Can you recommend some reading on this kind of topics? I certainly lack some basis.

Regards.

I know this in old thread, but I have similar questions / concerns. The explanation above certainly addresses the work 'pre-testing' using Zephyr...and is very similar to our approach.

But......it is the linking & tracking of story testing, via Zephyr which is a bit clunky. As it is not possible to see the Zephyr tests on the agile 'work' board and use should use the 'CYCLES' to organise the testing...it becomes cumbersome to tie the two streams up and provide adequate reporting (coverage of stores / tests etc). This is compounded by the limitation of 'assigning' tests to testers during a specific cycle.

Any suggestions would be welcomed. 

Should we have open/close statuses for test cases? Test cases are run on different cycles in different Sprints. If test cases are closed for the current iteration, then what should be their statuses for other test cycles?Thanks Mahnaz

I have a question regarding Test Cycle and Sprints in three different board.

I have one project with 3 Agile Boards and 3 sprints (one sprint in each board). How do I link the same test cycle to the sprints in the three agile boards. Currently I am getting a message saying: "The Cycle XYZ will be moved from Sprint PI3.I4 to PI3.I4-eSUB."

If I say Yes then then the Test Cycle is removed from one of the other boards. It looks like I am allowed to link the Test Cycle only to two Boards.

Can you please help with this issue? I need to link the Test Cycle to 3 sprints in three different boards. The Project is the same in all three boards.

Thanks

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published May 21, 2018 in Jira Software

How large do you think Jira Software can grow?

Hi Atlassian Community! My name is Shana, and I’m on the Jira Software team. One of the many reasons this Community exists is to connect you to others on similar product journeys or with comparabl...

1,221 views 10 18
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you