Comparison between Zephyr & Xray

[#1]

I am sharing this article with both Zephyr and Xray information and everything should be in one place. It will be easy to know when selecting Zephyr or Xray.

Zephyr and Xray are both Jira plug-ins used for test case management i.e., test case writing and execution purposes. Both do almost similar kinds of work with some advantages and disadvantages of them with respect to cost, automation, reporting, etc. 

Here is the feature level comparison for Zephyr and Xray

Reports:

Zephyr provides Traceability Report, Test Case Execution Report, Test Execution Burndown Chart and Top Defects Impacting Testing, etc. whereas Xray provides Traceability Report, Overall Requirement Coverage Report, Historical Requirement Coverage Report, Test Plans Report, Test Executions Report, Test Runs Report, etc.

Cost:

Reference Links:

Features: https://interworks.com.mk/comparison-of-jira-test-management-plugins/

Zephyr Advantage: https://www.getzephyr.com/insights/why-zephyr-jira-1-alternative-xray

Zephyr Details: https://zephyrdocs.atlassian.net/wiki/spaces/ZFJ0300/overview

Xray Advantage: https://www.getxray.app/xray-vs-zephyr/

Xray Details: https://confluence.xpand-it.com/display/public/XRAY/About+Xray

Zephyr Pricing: https://marketplace.atlassian.com/apps/1014681/zephyr-for-jira-test-management?hosting=server&tab=pricing

Xray Pricing: https://marketplace.atlassian.com/apps/1211769/xray-test-management-for-jira?hosting=server&tab=pricing

[Christian Pascher]

Regarding versioning:

The approach Xray is using is not really versioning that people expect. You cannot just select that you want to include version 2 into one test execution and version 3 of the same test in another test execution. Versioning is only supported in the way Jira is doing it.... But you definitely get a frozen list of test steps in a test execution that won't be changed if you change the test case afterwards. Which is most of the time enough, also for enterprises and big projects/programs from my experience. See https://docs.getxray.app/display/XRAY/Test+Versioning

Real versioning of test cases is only possible if you don't use Jira issues to store the date like it is done in for example in QMetry Test Management for Jira (https://marketplace.atlassian.com/apps/1215144/qmetry-test-management-for-jira?hosting=cloud&tab=overview)

[Taranjeet Singh]

This is very useful comparison of Xray and Zepyhr apps. Thanks for sharing this @Jayesh Wankhade !

[Jayesh Wankhade]

Gald that you liked it @Taranjeet Singh and helpful for you. Thanks for motivating me. 

[Piotr]

Hi, @Jayesh Wankhade  could you send me the snapshots from 'Zephyr vs Xray" comparison?

Unfortunately, I can't correctly displays Features 1 and 2 you posted.

 email: pwilkowski@o2.pl

[Jayesh Wankhade]

Thanks @Piotr piotr for pointing it out, and sorry for the late response. I have updated the images here as well and sent you on given email. Please let me know if you received it not.

[Piotr]

Thanks @Jayesh Wankhade for sharing your tool comparison sheet!

[Harsh Dubey]

@Jayesh Wankhade @Piotr This is outdated feature comparison list. ZSquad (brand name got updated as well) offers all the test management, automation related features.

Attaching the link for same: Feature-Link 

[Jayesh Wankhade]

Thanks, @Harsh Dubey for highlighting it. This blog was written a year back. 

[Sergejs Cuhrajs_SmartBear_]

Hi @Jayesh Wankhade

Would you mind updating the table? I think this is a useful resource for anyone researching more on Zephyr vs Xray, but some information is no longer accurate:

1. Zephyr for Jira is now called Zephyr Squad (thanks @Harsh Dubey for flagging this)

2. All table items under Automation section should be marked as 'Yes' (please see here for reference)

3. Built-in rest api - yes 

4. Jenkins and bamboo integration - yes

5. Number of reports – it should be 5 reports and 9 dashboard gadgets 

6. Pricing has also changed. I'd recommend visiting the Atlassian Marketplace for both apps for most up to date info

Kind Regards,
Sergejs from Zephyr team

[Jayesh Wankhade]

Thanks @Sergejs Cuhrajs_SmartBear_

I can update the table as per your recommendation and will be needing your help as well. If you are available let's connect and discuss more.

[Sergejs Cuhrajs_SmartBear_]

Hi @Jayesh Wankhade

Sounds great, you can reach out to me via email sergejs.cuhrajs[at]smartbear[dot]com

Thanks,
Sergejs

[Jayesh Wankhade]

Thanks @Sergejs Cuhrajs_SmartBear_ 

[Antonio Gil]

Hi @Sergejs Cuhrajs_SmartBear_ how does REST API use test versioning? I am not seeing any reference when retrieving the test at Zephyr Scale for Jira Cloud API (smartbear.com)  (getTestCase endpoint) neither when updating it (I mean, does update increase test case version by default) at Zephyr Scale for Jira Cloud API (smartbear.com) (udpateTestCase endpoint)

Can the active test version be configured? I mean, working in a new test version but meanwhile it is commited still using current version 

Related to both topics:  ... Does the jenkins plugin always access to last version? or to active version? 

[Sergejs Cuhrajs_SmartBear_]

@Antonio Gil hey there, so Zephyr Scale doesn't support versioning within our REST API, so every update goes to the latest version.

Regarding the Jenkins plugin – yes it will always use the latest version (as it also uses the API).

Hope this helps!

Kind Regards,
Sergejs

[Sally Makhlouf]

Hi @Jayesh Wankhade ,

Would you please send me the updated table ?

Many thanks,

Sally

[Kieron Southwood]

Hi @Jayesh Wankhade 

Sorry for jumping on the bandwagon here but are you able to send me the updated table as well? 

Thanks for the good work putting this together - its very helpful. 

Kieron

[Jayesh Wankhade]

Hello All

Sorry for the delayed, the article is updated now with all new information along with pricing details as well.

Thanks for your support.

Jayesh

[Jayesh Wankhade]

Hi @Sergejs Cuhrajs_SmartBear_ , Kieron Southwood,  Sally Makhlouf

It's been updated now.

[Gemerson]

Hi @Jayesh Wankhade 

Sorry for asking this, did you manage to also update the comparison table? Reading the comments it looks like the automation part it is the old version.

I am gathering information of differences and advantages between xray and zephyr. Which one do you prefer?

[Deepa Akinapally]

Hi,

I am currently using Zephyr Squad and the performance is terrible. 

Is anyone using X-Ray? How is the experience?

[Gemerson]

@Deepa Akinapally I am trying both in the trial mode. What are the problems you are having with zephyr?

On my side: Visually I prefer Xray, but at the performance side I prefer Zephyr Squad. To write the test steps in a test case with Xray cloud is a pain. It take a couple of seconds to load it and increases a lot the time to do anything.

[Deepa Akinapally]

We currently have Zephyr Squad, updating scripts like adding/modifying/ deleting is painfully slow(couple seconds).

Adding Test Scripts to Cycle is very painful sometimes. Can't get the right cycle to show and can't navigate to next pages. 

Are you saying X-Ray is not any better?

[Gemerson]

@Deepa Akinapally 
The problems I am having with xray is during the creation of new Test case steps, test plan or test set. It takes 2-3 seconds for each interaction to fully open the issue details. It is frustrating sometimes and even take longer than 3s for specific cases

However, to edit or delete it is okish. Do you have a way to ask to the jira admin to put the trial version of xray? That would be good to compare.

[Deepa Akinapally]

In Zephyr Squad, I changed IssueType = Test to IssueType = Test Migrated. I lost all the Test Details.

I am shocked. There was no mention of the consequences of the action.

Then I changed it back to IssueType=Test, I still didn't get my Test Details back.

Did anyone experience the issue that I am facing? 

[Harsh Dubey]

@Deepa AkinapallyIf you will change the issue type from 'Test' to any other issue type it will delete the test data. It's an expected behavior.
You can restore the backup (if you have) from the Admin -> General configuration section.

Regards,

Harsh