Comparison between Zephyr & Xray

I am sharing this article with both Zephyr and Xray information and everything should be in one place. It will be easy to know when selecting Zephyr or Xray.

Zephyr and Xray are both Jira plug-ins used for test case management i.e., test case writing and execution purposes. Both do almost similar kinds of work with some advantages and disadvantages of them with respect to cost, automation, reporting, etc. 

Here is the feature level comparison for Zephyr and Xray

1.PNG2.PNG

Reports:

Zephyr provides Traceability Report, Test Case Execution Report, Test Execution Burndown Chart and Top Defects Impacting Testing, etc. whereas Xray provides Traceability Report, Overall Requirement Coverage Report, Historical Requirement Coverage Report, Test Plans Report, Test Executions Report, Test Runs Report, etc.

Cost:

JIra pricing.png

Reference Links:

Features: https://interworks.com.mk/comparison-of-jira-test-management-plugins/

Zephyr Advantage: https://www.getzephyr.com/insights/why-zephyr-jira-1-alternative-xray

Zephyr Details: https://zephyrdocs.atlassian.net/wiki/spaces/ZFJ0300/overview

Xray Advantage: https://www.getxray.app/xray-vs-zephyr/

Xray Details: https://confluence.xpand-it.com/display/public/XRAY/About+Xray

Zephyr Pricing: https://marketplace.atlassian.com/apps/1014681/zephyr-for-jira-test-management?hosting=server&tab=pricing

Xray Pricing: https://marketplace.atlassian.com/apps/1211769/xray-test-management-for-jira?hosting=server&tab=pricing

34 comments

Christian Pascher
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
October 16, 2020

Regarding versioning:

The approach Xray is using is not really versioning that people expect. You cannot just select that you want to include version 2 into one test execution and version 3 of the same test in another test execution. Versioning is only supported in the way Jira is doing it.... But you definitely get a frozen list of test steps in a test execution that won't be changed if you change the test case afterwards. Which is most of the time enough, also for enterprises and big projects/programs from my experience. See https://docs.getxray.app/display/XRAY/Test+Versioning

Real versioning of test cases is only possible if you don't use Jira issues to store the date like it is done in for example in QMetry Test Management for Jira (https://marketplace.atlassian.com/apps/1215144/qmetry-test-management-for-jira?hosting=cloud&tab=overview)

Like # people like this
Taranjeet Singh
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
December 6, 2020

This is very useful comparison of Xray and Zepyhr apps. Thanks for sharing this @Jayesh Wankhade !

Jayesh Wankhade December 6, 2020

Gald that you liked it @Taranjeet Singh and helpful for you. Thanks for motivating me. 

Piotr May 26, 2021

Hi, @Jayesh Wankhade  could you send me the snapshots from 'Zephyr vs Xray" comparison?

Unfortunately, I can't correctly displays Features 1 and 2 you posted.

 email: pwilkowski@o2.pl

Like Jayesh Wankhade likes this
Jayesh Wankhade June 10, 2021

Thanks @Piotr piotr for pointing it out, and sorry for the late response. I have updated the images here as well and sent you on given email. Please let me know if you received it not.

Like Piotr likes this
Piotr June 11, 2021

Thanks @Jayesh Wankhade for sharing your tool comparison sheet!

Like Jayesh Wankhade likes this
Harsh Dubey June 11, 2021

@Jayesh Wankhade @Piotr This is outdated feature comparison list. ZSquad (brand name got updated as well) offers all the test management, automation related features.

Attaching the link for same: Feature-Link 

Like # people like this
Jayesh Wankhade June 11, 2021

Thanks, @Harsh Dubey for highlighting it. This blog was written a year back. 

Like Harsh Dubey likes this
Sergejs Cuhrajs_SmartBear_
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
June 24, 2021

Hi @Jayesh Wankhade

Would you mind updating the table? I think this is a useful resource for anyone researching more on Zephyr vs Xray, but some information is no longer accurate:

1. Zephyr for Jira is now called Zephyr Squad (thanks @Harsh Dubey for flagging this)

2. All table items under Automation section should be marked as 'Yes' (please see here for reference)

3. Built-in rest api - yes 

4. Jenkins and bamboo integration - yes

5. Number of reports – it should be 5 reports and 9 dashboard gadgets 

6. Pricing has also changed. I'd recommend visiting the Atlassian Marketplace for both apps for most up to date info

 

Kind Regards,
Sergejs from Zephyr team

Like # people like this
Jayesh Wankhade June 24, 2021

Thanks @Sergejs Cuhrajs_SmartBear_

 

I can update the table as per your recommendation and will be needing your help as well. If you are available let's connect and discuss more.

Sergejs Cuhrajs_SmartBear_
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
June 24, 2021

Hi @Jayesh Wankhade

Sounds great, you can reach out to me via email sergejs.cuhrajs[at]smartbear[dot]com

 

Thanks,
Sergejs

Like Jayesh Wankhade likes this
Jayesh Wankhade June 25, 2021
Antonio Gil September 8, 2021

Hi @Sergejs Cuhrajs_SmartBear_ how does REST API use test versioning? I am not seeing any reference when retrieving the test at Zephyr Scale for Jira Cloud API (smartbear.com)  (getTestCase endpoint) neither when updating it (I mean, does update increase test case version by default) at Zephyr Scale for Jira Cloud API (smartbear.com) (udpateTestCase endpoint)

Can the active test version be configured? I mean, working in a new test version but meanwhile it is commited still using current version 

Related to both topics:  ... Does the jenkins plugin always access to last version? or to active version? 

Sergejs Cuhrajs_SmartBear_
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
September 9, 2021

@Antonio Gil hey there, so Zephyr Scale doesn't support versioning within our REST API, so every update goes to the latest version.

Regarding the Jenkins plugin – yes it will always use the latest version (as it also uses the API).

Hope this helps!

Kind Regards,
Sergejs

Sally Makhlouf October 26, 2021

Hi @Jayesh Wankhade ,

Would you please send me the updated table ?

Many thanks,

Sally

Like Jayesh Wankhade likes this
Kieron Southwood March 8, 2022

Hi @Jayesh Wankhade 

 

Sorry for jumping on the bandwagon here but are you able to send me the updated table as well? 

Thanks for the good work putting this together - its very helpful. 

Kieron

Like Jayesh Wankhade likes this
Jayesh Wankhade March 8, 2022

Hello All

Sorry for the delayed, the article is updated now with all new information along with pricing details as well.


Thanks for your support.

Jayesh

Like # people like this
Jayesh Wankhade March 8, 2022

Hi @Sergejs Cuhrajs_SmartBear_ , Kieron Southwood,  Sally Makhlouf

It's been updated now.

Like # people like this
Gemerson March 25, 2022

Hi @Jayesh Wankhade 

 

Sorry for asking this, did you manage to also update the comparison table? Reading the comments it looks like the automation part it is the old version.

I am gathering information of differences and advantages between xray and zephyr. Which one do you prefer?

Like maree_milne-v likes this
Deepa Akinapally April 4, 2022

Hi,

I am currently using Zephyr Squad and the performance is terrible. 

Is anyone using X-Ray? How is the experience?

Gemerson April 4, 2022

@Deepa Akinapally I am trying both in the trial mode. What are the problems you are having with zephyr?

On my side: Visually I prefer Xray, but at the performance side I prefer Zephyr Squad. To write the test steps in a test case with Xray cloud is a pain. It take a couple of seconds to load it and increases a lot the time to do anything.

Deepa Akinapally April 5, 2022

We currently have Zephyr Squad, updating scripts like adding/modifying/ deleting is painfully slow(couple seconds).

Adding Test Scripts to Cycle is very painful sometimes. Can't get the right cycle to show and can't navigate to next pages. 

Are you saying X-Ray is not any better?

Gemerson April 6, 2022

@Deepa Akinapally 
The problems I am having with xray is during the creation of new Test case steps, test plan or test set. It takes 2-3 seconds for each interaction to fully open the issue details. It is frustrating sometimes and even take longer than 3s for specific cases

However, to edit or delete it is okish. Do you have a way to ask to the jira admin to put the trial version of xray? That would be good to compare.


Deepa Akinapally April 22, 2022

In Zephyr Squad, I changed IssueType = Test to IssueType = Test Migrated. I lost all the Test Details.

I am shocked. There was no mention of the consequences of the action.

Then I changed it back to IssueType=Test, I still didn't get my Test Details back.

Did anyone experience the issue that I am facing? 

Like Jayesh Wankhade likes this
Harsh Dubey April 25, 2022

@Deepa AkinapallyIf you will change the issue type from 'Test' to any other issue type it will delete the test data. It's an expected behavior.
You can restore the backup (if you have) from the Admin -> General configuration section.

Regards,

Harsh

Like Jayesh Wankhade likes this

Comment

Log in or Sign up to comment
TAGS
AUG Leaders

Atlassian Community Events