Testing apps for Jira comparison: QAlity Plus vs Xray vs Zephyr Squad

[#1]

In our recent series of articles, we’ve compared 3 popular testing apps for Jira - QAlity Plus, Xray, and Zephyr Squad. We’ve discussed the similarities and differences in test writing, test cycles, test executions, and reporting, showcasing the capabilities of each product. Today, we would like to summarize the series and provide an overview of each app’s offer.

Test writing

The differences between test writing-related features in the 3 apps are predominantly the matter of interface. Most of these differences do not have any significant impact on the core functionalities such as test creation, editing, cloning, or deleting, which are present in each solution, and work in a similar manner. There are, however, some distinct characteristics worth mentioning in order to paint the whole picture of how each app approaches the test writing topic.

Key points of comparison:

Below, you will find a list of the crucial insights discussed in the complete comparison article.

Creating tests

Creating tests by choosing a test issue type: all 3 apps

Creating tests directly in an existing issue: QAlity Plus

Creating test steps

Adding test steps at the end of the test steps list: all 3 apps

Adding test steps in between already created steps, on a chosen position: Xray

Viewing options

Offering two different viewing options, one more detailed and the other simplified: all 3 apps

Collapsing or expanding the test steps view: QAlity Plus, Xray

Adding additional columns and customizing the elements that are to be shown: Zephyr Squad

Editing

All elements edited inline: QAlity Plus

Type and definition edited inline, test steps edited in an edit mode: Xray

All elements edited in an edit mode: Zephyr Squad

Reordering

Reordering steps by dragging and dropping: all 3 apps

Manually setting the position for a test step: Xray

Cloning and deleting

Cloning and deleting feature available in test issue menu: all 3 apps

Cloning and deleting test steps by clicking on action icons: Xray, Zephyr Squad

If you need a detailed comparison, we’ve covered the topic in this article: Test writing in QAlity Plus, Xray and Zephyr Squad 

Test cycles

Test organization introduces more differences between QAlity Plus, Xray, and Zephyr Squad. Each app offers its own unique system of grouping test cases. Depending on the user’s needs, different approaches to test cycles can turn out to be the most suitable.

Key points of comparison:

Below, you will find a short description of the QAlity Plus, Xray, and Zephyr Squad’s offer, highlighting the most distinct points for each.

Overview

QAlity Plus

Test Cycle Overview page provides a comprehensive view of your test cycles, offering essential details such as project, version it is assigned to, status of test executions, creation date, and due date. It also allows filtering of test cycles based on both project and version criteria.

Xray

Tests can be organized into Test Plans, instead of test cycles. Test Plans provide an insight into the most crucial information on the related tests: a list of tests, their executions, dataset, latest status as well as the overall execution status. Additionally, you can group tests in Test Sets or Test Repositories. 

Zephyr Squad

Test Cycle Summary page displays the details about your test cycles, depending on the selected version of the project. Basic information includes status, summary, start date, end date, and more. You can also modify the columns displayed in your Test Cycle Summary.

Creating test cycles

Creating test cycles from the test cycles page: QAlity Plus, Zephyr Squad

Creating a test plan by choosing a test plan issue type: Xray

Reordering test cases within test cycles

Reordering test cases by dragging and dropping: all 3 apps

Manually setting the position for a test case: Xray

If you need a detailed comparison, we’ve covered the topic in this article: Test Cycles in QAlity Plus, Xray and Zephyr Squad

Folders and Repository

Organizing test items is also possible in folders. They introduce more flexibility into your test management, allowing you to group test items according to your needs. Folders can be used to segregate items related to different modules of your application. Another common use case is to organize test cases in folders to easily add them to test cycles later. All 3 apps provide folder-related functionality. Each one of them also gives you access to a Repository, where you can view and manage your items.

Key points of comparison:

Below, you will find a list of the crucial characteristics and differences between the apps’ offer.

Creating folders: all 3 apps

Adding to folders test cases: QAlity Plus

Adding to folders tests and preconditions: Xray

Adding to folders test cases, test plans and test cycles: Zephyr Squad

Adding to a folder items from multiple projects: QAlity Plus

Adding one item to multiple folders: QAlity Plus

Creating sub-folders: Xray, Zephyr Squad

Test Case Repository: QAlity Plus, Zephyr Squad

Test Repository: Xray

Test Execution

The core part of testing is, of course, test execution. The structure and interface of this feature can significantly impact the comfort and efficiency of the tester’s work.

Key points of comparison:

Below, you will find a short summary of the QAlity Plus, Xray, and Zephyr Squad’s offer, with the key differences listed.

Starting the execution process

Ad hoc or as a part of Test Cycle or Test Plan: all 3 apps

By creating a test execution issue type: Xray

General info on the execution page

QAlity Plus: execution assignee, description, comments

Xray: assignee, executed by, versions, revision, dates, timer, environments, findings, test details

Zephyr Squad: status, assignee, summary, description, when last performed and by whom, defects, comments

Assigning users

QAlity Plus: on the Test Cycle page or within a specific Test Case Execution

Xray: creating separate test execution issues for each tester or assigning runs within test execution issue based on specific user type fields

Zephyr Squad: on the Test page or Test Cycles page

Recording results

Available statuses:

QAlity Plus: Unexecuted, In Progress, Passed, Failed, Blocked

Xray: Todo, Executing, Pass, Fail + Custom

Zephyr Squad: Unexecuted, WIP, Pass, Fail, Blocked + Custom

Overall execution status

Execution status is what informs the user about the current result of testing. Different apps approach naming and updating statuses differently. Each solution is also characterized by its own overall execution status update requirements.

Overall execution status:

QAlity Plus: If all tests are passed, overall execution status will automatically be set to passed as well.

Xray: Overall execution status is automatically adjusted to the current status of test executions.

Zephyr Squad: Overall execution status does not update automatically, and can only be changed manually.

Reporting defects

QAlity Plus: linking or creating bugs directly from the Test Execution page; bug description generated automatically (the executed test case name, execution date, link to the execution, and test case steps)

Xray: linking or creating bugs in the Findings section of your Test Execution; linking or creating bugs for specific Test Steps

Zephyr Squad: linking or creating bugs for either the Test Execution or the specific Test Steps

Adding attachments

Adding attachments to specific Test Steps: all 3 apps

Adding attachments to the whole Test Execution: Xray, Zephyr Squad

Execution history

QAlity Plus: available within the current Test Execution; includes information such as how many times the test case has been executed already, the date and status of the previous execution, and the date when the test case last failed; consists of two tabs - Bugs reported and Steps failed

Xray: available in the Test Runs section of the Test issue page

Zephyr Squad: provides information such as status changes, attachment actions (add/update/remove), assignee modifications, comment actions (add/update/remove), and custom field adjustments

Test case versioning

QAlity Plus: during test execution, the current version is locked and cannot be modified

Xray: creating and handling different versions of a test within the same Test issue

Zephyr Squad: no test case versioning

If you need a detailed comparison, we’ve covered the topic in this article: Test Executions in QAlity Plus, Xray and Zephyr Squad

Reporting

There would be no use for testing without functional reporting features in place, providing the user with an insight into the results of test execution and the overall condition of the tested software.

Reports exporting features in each app:

Below, you will find a rundown of the QAlity Plus, Xray, and Zephyr Squad’s offer, focusing on the reporting options available in each app.

Test Execution Reports

QAlity Plus: report of all test executions for a chosen period of time; two additional views available - status chart and assignee chart; can be exported to an .XLSX file

Xray: in-built reports for particular pages (Tests List Report, Test Sets List Report, Test Executions List Report, Test Plans List Report, Test Runs List Report, Test Plans Metrics Report); can be exported to a CSV file

Zephyr Squad: Test Metrics (Daily Test Execution Progress by status, Test Executions by Test Cycle, Test Executions by Tester, Number of executions per day, List of executions with their details); reports for specific projects (Test Execution Chart, Test Execution Burndown Chart, Top Defects Impacting Testing Report), no exporting available

Coverage & Traceability Reports

QAlity Plus: Coverage and Traceability combined into one report, no exporting available

Xray: two separate reports for Coverage and Traceability; Traceability Report can be exported to a CSV file

Zephyr Squad: Traceability Matrix, with forward and backward reports available; can be exported to an Excel file

If you need a detailed comparison, we’ve covered the topic in this article: Reporting in QAlity Plus, Xray and Zephyr Squad

Cloud Pricing

Here are the monthly cloud prices for each of the three apps, shown in three examples - for up to 10 users, for 50 users, and for 1000 users:

Summary

Testing plays a significant role in any development process. For that reason, there are countless testing apps for Jira available on the Atlassian Marketplace. Examining the differences between each solution can be a laborious and time-consuming task. To make it less daunting, we’ve explored the offer of 3 popular testing apps - QAlity Plus, Xray, and Zephyr Squad, and compiled the unique features and differentiating characteristics of each.

Our aim was to construct a comprehensive overview of the available options, to provide the fuller picture of their approach to the topic of testing.

The comparison series revealed both small and big differences between the apps. The details were already discussed extensively in the previous articles of the series. Generally speaking, each app has its own points of focus and can answer the needs of different users. 

QAlity Plus is the simplest out of 3. Being true to the “less is more” phrase, it puts the attention on intuitiveness, aiming to interact with the user in a way that is easy to understand. At the same time, it provides all the fundamental testing features that the other 2 apps also have. What QAlity Plus lacks, is some of the advanced functionalities that would make the app a lot more complex and less user-friendly. When it comes to pricing, QAlity Plus is the most cost-efficient option, being 2 to 3 times less expensive than Xray and Zephyr Squad.

Xray, in terms of complexity, is the opposite of QAlity Plus. It offers advanced, multi-layered features that require a lot of time and focus to learn, but can be very useful in organizing tests and executing them. Xray puts a lot of effort into creating a unique system. It makes the app less intuitive than QAlity Plus, but enables more complex operations. As for the pricing, Xray is the only app out of 3 that requires payment even for a 10 users instance. The overall pricing is higher than QAlity Plus, but lower than Zephyr Squad.

Zephyr Squad, compared to the other 2 apps, allows more customization. The user can modify the view according to their needs by adding or removing columns that are to be displayed. This interactive approach gives the user a chance to decide what information is the most crucial to them, and what should stay hidden to not obstruct the view. Zephyr Squad is the most costly option out of the 3 apps compared in this series.

Each app has its own strengths and weaknesses. We hope that this comparison series helped you to determine which of the 3 solutions meets your requirements, and suits your team best.