Executing tests in QAlity Plus, Zephyr, and XRay: app comparison

[#1]

In the previous articles, we’ve compared test writing features and test case grouping and organization in three test management apps for Jira - QAlity Plus, Xray, and Zephyr Squad. In this article, we shift our focus to the very heart of the testing process - test execution. Our main aim is to show how each app handles test execution and what features their execution pages offer. We'll also include price references for each app, helping you make an informed decision based on the features you'll get for your investment.

Starting the execution process

In all three apps, you can perform an ad-hoc execution as well as execute tests as a part of a test cycle or test plan. 

QAlity Plus

You can start the test execution from the test cycle page by clicking ‘Open execution’ under ‘Actions’. You will then be taken to the execution page. Alternatively, you can click on the test case menu and select ‘Execute’.

Then, a modal will appear where you can choose whether you want to execute the test case ad-hoc or as a part of a test cycle:

After making the choice and clicking ‘Execute’ you will be transferred to the execution page:

Xray
You can execute tests directly from the test plan, perform ad-hoc test executions directly from a test, or create a new Test Execution issue.

The execution will start when you click ‘Execute’ (play button):

You will then be taken to the Test Execution screen: 

Zephyr Squad 

You can execute a test ad-hoc, from the Test Cycle Summary page, or from the Test Executions section of a test. In all these scenarios, there will be an ‘E’ button you need to click.

This will start the execution process and you will be taken to the execution page:

General information about the execution that can be found on the execution page

In all three apps, you'll find essential information concerning the particular execution at the top of the execution page, including the execution status. However, the specific details included in this section vary among the apps.

QAlity Plus

You can see the name of the test case and the Execution Assignee (or if the test is unassigned, you will be able to assign it to a user who is supposed to perform it). You will also find the Description field that contains information from the test case concerning this execution. Another field is the Execution Comment where you can provide any relevant information or observations during the execution process, including the test outcomes. 

Xray

Next to the Execution Status, you will find fields such as:

• Assignee -  the user assigned to perform the current test execution
• Executed By - the last user that changed the status of the current test run
• Versions - the target release version tested by the current test execution
• Revision - the source code and documentation version used in the current test execution
• Started On - the date and time the execution of the current test started
• Finished On - the date and time the execution of the current test finished
• Timer which tracks the execution time
• Test Environments - the testing environment of the execution

Furthermore, you'll find dedicated sections like 'Findings' and 'Test Details,' where you can add comments, defects, attachments, Test Issue links, and preconditions for the entire test execution. These sections serve as a centralized hub for storing all relevant information associated with the overall execution process.

Zephyr Squad 

Similarly to the two previous apps, the section serves as a central point to manage the execution information. It contains the Execution Status, Assignee (you can also assign a user here), Summary, and Description, as well as the information on when the execution was last performed and by whom. You can also add defects and comments related to the whole execution. 

Overall Execution Status

The Overall Execution Status represents the current state of the entire execution process. While this aspect is present in all three apps, there are slight variations in how it functions within each of them.

QAlity Plus
You can set the Execution Status to one of the following options: In Progress, Passed, Failed, Blocked, and Unexecuted.
If all test steps have the status 'Passed,' the Execution Status will automatically change to 'Passed.' However, for any other scenarios, you need to manually set the appropriate status.

Xray
The native statuses you can set are: TODO, EXECUTING, FAIL, and PASS, with the option to create custom statuses. 

The Overall Execution Status automatically adapts based on the statuses assigned to individual test steps during the execution process. 

Furthermore, in addition to being displayed at the top of the Execution page, the Overall Execution Status in Xray is also visualized within the corresponding Jira issue as a colored progress bar."

Zephyr Squad
Default options for the Execution statuses, include UNEXECUTED, PASS, FAIL, WIP (Work in Progress), and BLOCKED. Additionally, they can be customized by your Jira Administrator. 

Unlike in Xray and QAlity Plus, in the case of ‘Passed’, the Execution Status of the overall test in Zephyr Squad is not automatically updated based on the status of individual test steps - you have to set it manually.

Assigning users
By designating specific users to test executions or test cases, teams can effectively divide responsibility for particular testing tasks.

QAlity Plus
You can assign a user to a test execution either on the Test Cycle page or directly on the specific test case execution. It’s also possible to unassign the user from the test case by clicking on the x button next to their name.

Xray
You can either create separate Test Execution issues for each Tester or assign Test Runs within a Test Execution issue based on specific user type fields. You can also bulk assign all the selected Test Runs to a particular user.

 

Zephyr Squad
You can assign a particular test to a user while viewing or editing the test. Another way is to assign tests from test cycles: once test cycles are created or cloned, and tests are added to them for execution, you can assign the test executions to various users. 

Additionally, there’s the "Saved Filters" option that allows you to create custom filters displaying only a particular user's test execution assignments. This can be achieved by modifying the ZQL (Zephyr Query Language) of the test cycle and saving it as a favorite, private, or global filter.

Recording results
Recording results involves documenting the outcomes of test executions. Testers achieve this by setting appropriate statuses to each of the test steps.

QAlity Plus
You can record results for each step with one of the following statuses: In Progress, Passed, Failed, Blocked, and Unexecuted, which is set as the default status. To efficiently manage multiple statuses, users can make bulk changes which allow for modifying the status of several steps at once.

Xray
Test steps’ statuses can be set to: TODO, EXECUTING, FAIL, or PASS. Each test step is visually represented with a corresponding color based on the status you assign

Additionally, Xray allows Inline Test Execution which gives you the ability to execute test runs directly from the Test Execution view screen, without having to go through the traditional process of executing all the test steps manually, one by one.

When executing inline tests, the status of the manual steps can be changed automatically based on the following rules:

• PASS: If you select this status, all the test steps will be automatically marked as passed.
• FAIL: If you choose this status, all step statuses will remain unchanged, even if some or all of them are marked as PASS. This is because a test might fail due to a precondition not being met, even if the actual steps were executed successfully.
• TODO: When you select this status, all step statuses will be changed to TODO.
• EXECUTING: the test execution status will be changed to EXECUTING, which signifies that the test run is currently in progress.
• CUSTOM: This option allows you to set a custom status for the test execution.

Zephyr Squad
You can assign one of the following statuses to test steps: UNEXECUTED, PASS, FAIL, WIP (Work in Progress), and BLOCKED. Additionally, similar to the Execution Status, Zephyr Squad enables users to set custom statuses for specific test steps.

Moreover, the app offers the Quick Execute feature allowing you to change the status of a test directly from the Test Cycle list or while viewing an issue, without the need to open each test individually.

Reporting defects
Once defects have been found during the execution, they need to be appropriately documented and communicated, in order to provide crucial information for developers to understand and address the issues. 

QAlity Plus
You can link existing bugs and create new ones directly from the test execution page by clicking the plus button located in the test step that has an issue and selecting a bug from the list. If you need to create a new bug, you can do it by clicking ‘+Create New’. When creating a new bug, the system will automatically generate a description that includes crucial information such as the executed test case name, execution date, link to the execution, and test case steps, which saves testers from having to enter the details manually.

The bugs you’ve selected will be automatically added to the linked issues for the given step in the current execution. 

When you hover over the number of the linked bugs, you will see a pop-up with a detailed list of the bugs that have been linked to that particular step.

Xray
You can add bugs (defects) to the whole execution, under the Findings section. You can create a new defect or sub-defect as well as add existing ones. 

In a similar manner, you can create a new defect or sub-defect or add existing ones to particular test steps.

Zephyr Squad 
Similarly to Xray, you can create a new defect or link an existing bug to both: the test execution and a particular test step.

Adding Attachments 

QAlity Plus

The app does not provide the option to add attachments directly to the entire test execution on the execution page. There is the option to add attachments to particular test steps though. You can upload them by clicking the button, or you can use a drag and drop. 

 

Xray

You can add attachments to the entire test execution by clicking the plus button and selecting 'Add Evidence' within the 'Findings' section on the Execution page.

Furthermore, you have the option to add attachments to specific test steps in a similar manner by clicking 'Evidence' under the respective step. 

Zephyr Squad
Files can be attached on the execution page to both the whole execution and particular test steps. Other users can preview the attachment(s) before actually downloading the file.

Execution history

All three apps contain execution history where you can find information about past executions.

QAlity Plus

During test execution, you can view past executions’ statuses and bugs linked to failed steps in past executions. You can compare current and past results, without the need for additional searching, as the execution history is available within the current execution. Previous execution history will tell you how many times the test case has been executed already, the date and status of the previous execution, and the date when the test case last failed.

The previous execution history consists of two tabs. The first one, ‘Bugs reported’, displays a list of all bugs that have been linked to failed steps in the previous executions. Additionally, you will find there information such as the status of the issue, reporter, and creation date. 

The second tab, failed steps, will show you all failed steps from previous executions with additional information such as bugs linked, comments, and test cycle.

Zephyr Squad
The Execution History provides a chronological record of all changes made to this test execution, including details of who made the changes and when. It logs various operations on test executions, such as status changes, attachment actions (add/update/remove), assignee modifications, comment actions (add/update/remove), and custom field adjustments.

Xray

You can access the execution history of a specific Test issue within the "Test Runs" section of the Test issue screen. This section provides a comprehensive view of the history of previous test runs for that particular Test.

Test case versioning

Test case versioning involves creating and managing multiple iterations of a test case. Each version captures a specific modification or update to the test case, resulting in a clear history of changes over time. 

QAlity Plus

QAlity Plus consistently shows the latest version of the QAlity Test issue type (Test case). However, during test execution, the version for that particular execution is locked. To include any modifications, you must execute the test case again. You have two options: either clone the test cycle to create a new, fresh cycle without any executions, or delete the execution of the edited test case and then re-execute it.

Xray

With Xray version 7.0 and later, you can create and handle different versions of a test within the same Test issue, so you can keep track of changes and revert to specific versions when needed. When you schedule tests for execution in Test Runs, Xray creates copies of the test specifications to maintain data consistency. Any updates to the test specifications won't affect previously recorded Test Runs unless you specifically want them to.

Zephyr Squad

The app does not offer test case versioning. When a test case is updated, changes made to already executed test cases become visible in previous test execution records. So, for example, a step added to a test case will show up in a historic test execution as an unexecuted step.

Price Comparison

Below you can compare monthly cloud prices for each of the three apps mentioned in this article, for up to 10 users, for 50 users, and for 1000 users:

QAlity Plus   

Up to 10 users free

50 users = 75.50$/mo

1000 users = 570$/mo

Xray 

10 users = 10$/mo

50 users = 212.50$/mo

1000 users = 1,362.50$/mo

Zephyr Squad

Up to 10 users free

50 users = 255.50$/mo

1000 users = 1,874.50$/mo

Summary

This article is an examination of the test execution process in three testing apps: QAlity Plus, Zephyr Squad, and Xray. The comparison is based on their execution pages, with a focus on essential features such as setting the test step and overall execution status, comment and attachment capabilities, execution history, and test versioning. Additionally, the article includes price references for each app, in order to help you with your purchase decision by presenting the features offered in relation to their cost.