Reporting in QAlity Plus vs Xray vs Zephyr Squad

[#1]

In our previous articles, we’ve already discussed test creation, test cycles and test execution-related features in 3 popular testing apps for Jira - QAlity Plus, Xray, and Zephyr Squad. The series wouldn’t be complete without exploring reporting - one of the most important elements of any testing app.

Reports are what joins all the pieces of information together, delivering a comprehensive insight into your testing as well as its results. They also allow viewing the gathered data from different angles, creating the basis for deep, all-encompassing analysis. Depending on the reporting features offered by testing software, different levels and types of analysis can be achieved. For that reason, it is important to take into account reporting options available in testing apps when deciding which solution works best for you.

Reporting Overview

Each of the apps for Jira discussed in this article offers its own set of reporting features. They differ in functionality as well as the interface, making them suitable for different users, depending on their needs and preferences. Below, you will find detailed information about the most important reporting features offered by each of the 3 apps, grouped into two main categories - Test Execution Reports and Coverage & Traceability Reports.

Test Execution Reports

This section is dedicated to the reporting features focused on tracking your test executions and their results.

QAlity Plus

You can view all of the tests executed in a selected date range, along with the test cycle that they belong to, execution date, assignee, and the result. The executions can be filtered by project, user, and status.

 

To access the page, enter the QAlity Plus app and click on the Test Execution Report in the Reports section. You can also access it via your project menu. To do that, go to your project menu and choose the Reports section. Scroll down to Other and click on the Test Execution Report.

The standard table view can be replaced by a status chart or assignee chart.

 

Status chart view, showing the ratio of test executions statuses:

 

Assignee chart view, showing the statuses of tests executed, grouped by the users that executed the tests:

To export this report, click on the Export button in the top right corner of the page. All records, matching the filters, will be exported to an XLSX file.

Xray

Your test executions can be viewed through build-in reports available on the particular pages: 

• Tests List Report: allows you to view a list of Test issues and their execution status for a chosen scope.

• Test Sets List Report: allows you to view a list of Test Set issues and their execution status, for a chosen scope.

• Test Executions List Report: allows you to view a list of Test Execution issues and their execution status.

• Test Plans List Report: allows you to view a list of Test Plan issues and their execution status.

• Test Runs List Report: allows you to view the current status of tests, their results, and other information, such as dates of execution, user assigned to execution, test type, execution version, etc.)

• Test Plans Metrics Report: allows you to view your Test Plan metrics, including tests by status and success rate bars, as well as the total number of test executions, test runs, and tests, within the Test Plan.

To access the reports listed above, click on the Testing Board icon on the project left sidebar, or on the Reports icon in your Jira menu.

All of the mentioned reports, except for the Test Plans Metrics Report, can be exported to the CSV file. To export the report, click on the Export button in the top right corner of your issue.

Zephyr Squad

There are a few ways to track the progress of your test executions. You can either use test metrics or access the reports available for specific projects.

To view test metrics, select Zephyr in your Jira menu and choose Test Metrics in the sidebar. With this feature, you can track the following:

• Daily Test Execution Progress by status,
• Test Executions by Test Cycle,
• Test Executions by Tester,
• Number of executions per day,
• List of executions with their details.

You can customize the date range of the displayed information as well as choose the version of your project. Panels can be moved around the page by dragging and dropping. All changes made to the Test Metrics page are refreshed once the current session is over.

You can also monitor your test executions in the reports available for every project. It can be viewed by choosing Reports in your Jira menu, and then scrolling down to the Other section.

 

There are 3 options available:

• Test Execution Chart: allows you to view execution information for the selected project and version in the form of a bar graph. 

 

You can edit the contents displayed in your Test Execution Chart by clicking on the three-dot menu in the top right corner. Data can be grouped by Component, User or Cycle. You can also change the title, project and version, as well as modify the Refresh Interval:

• Test Execution Burndown Chart: allows you to view cycle execution information on a timeline for the selected project and version.

 

Similarly, you can edit the chart by clicking on the three-dot menu in the corner.

 

• Top Defects Impacting Testing report: allows you to view a list of the defects which are blocking the biggest number of tests from passing.

 

You can customize the chart by clicking on the three-dot menu in the corner.

Coverage & Traceability Reports

The reports that play an essential role in analyzing your testing results and the quality of your software are dealing with test coverage and traceability. Coverage reports provide you with an insight into how your testing covers the specific parts of your application, and help to identify the spots that require more attention. Traceability reports are linking the information on tests, their executions, and the bugs discovered, combining them into a comprehensive record of software’s condition, and allowing you to make informed decisions on further testing and development.

In the following section, we will explore the reporting features related to coverage and traceability.

QAlity Plus

The Traceability Report in QAlity Plus involves both elements mentioned above. It allows you to visually assess the test coverage in your project, as well as track the connection between the tests, test executions, and bugs over the selected time period, and in a selected project to diagnose prevailing issues that affect the quality of your system.

The report provides you with an insight into the project in terms of how many times specific test cases were executed, what the result of the latest execution was, and bugs that were created while executing test cases in the selected date range. You can also check the linked bugs by expanding the list to see the issues' status.

 

Requirement coverage can be viewed once you select the proper issue type, e.g. Epic or Task.

 

The report can be filtered to display only active test cases.

 

You can access the Traceability Report by selecting the Reports button in your Jira project menu, scrolling down to view the Other section, and clicking on the Test Traceability Report. It can also be found on the QAlity page in your Apps.

There is no exporting feature for this report.

Xray

Xray offers two separate reports for test coverage and traceability.

Test Coverage Report

Provides a visual view of the test coverage status presented in the form of a bar chart.

 

You can choose the analysis scope, group and filter your results, and modify the view.

The button More allows you to add additional filters.

To access this report, select Xray icon on the left sidebar of your project. You can also head to Reports in your Jira menu, and find it in the section Other.

There is no exporting feature for this report.

Traceability Report

Provides a comprehensive insight into the requirements traceability, combining the data on tests, test runs and bugs. It can be viewed on either Epic, Stories or Sub-tasks level. The report allows you to examine the issue from both forward and backward direction. 

To access this report, select Xray icon on the left sidebar of your project. You can also head to Reports in the Jira menu, and find it in the section Other.

Traceability Report can be exported to the CSV file. To export the report, click on the Export button in the top right corner of your issue.

 

Zephyr Squad

Tracking the relationship between your tests, test executions, requirements and bugs can be achieved in the Traceability Matrix, where you’re allowed to generate both forward and backward traceability reports.

 

To create a Traceability Report, select Traceability Matrix on the left sidebar. You can choose the type of your report by clicking on the Report Type button in the top right corner.

 

The report can be adjusted to your needs with primary (requirement type, versions, cycles, folders, execution status) and secondary (requirements without tests, executions without defects, executions with defects) filters.

 

Traceability Report can be exported to Excel. To do that, click on the Export button in the top right corner. 

 

The file will consist of 4 columns: Requirements, Tests, Executions, and Defects.

Cloud Pricing

Although we’ve already mentioned pricing in our previous articles from this series, it is such an important part of choosing the app suitable for your needs that it cannot be omitted in this installment as well. The 3 apps compared in this article are all providing tools to generate a few types of reports. The aim is to equip the user with comprehensive information on the state of testing and the condition of their software. Depending on whether you need forward and backward issue tracking, or whether you prefer separate or combined reports for coverage and traceability, different solutions will be most suitable for you. Reporting in each of these apps is uniquely designed and organized, making the choice of the best option mostly a matter of subjective preference. The factor that can help in this decision is the price.

Here are the monthly cloud prices for each of the three apps, shown in three examples - for up to 10 users, for 50 users, and for 1000 users:

QAlity Plus   

Up to 10 users free

50 users = 75.50$/mo

1000 users = 570$/mo

Xray 

10 users = 10$/mo

50 users = 212.50$/mo

1000 users = 1,362.50$/mo

Zephyr Squad

Up to 10 users free

50 users = 255.50$/mo

1000 users = 1,874.50$/mo

Summary

The above comparison can serve as a reference for considering the available options of testing apps, their unique features and user interfaces, as well as their cost. It is focused on the reporting-related functionalities only, with the purpose of zooming in on the details and nuances of each vendor’s offer. In the previous articles, we’ve already compared other areas of testing apps’ features, such as test writing, test cycles and test execution, to provide a full picture and, hopefully, make it easier for you to decide on the solution most suitable for your needs. The final installment of the series will summarize the contents of all previous comparisons.