Better Addon Xray or Zephyr?

priya abuj November 14, 2018

Hello,

I would like to know is it good idea to use Xray addon instead of Zypher.

It will really help me to get answers from real users of these addons.

Currently I am using trial version of Xray to see what is beneficial in Xray.

Few things I liked

1.  you can write and manual and automation test cases

2. Xray Reports are really user friendly.

 

Thank you 

Priya

5 answers

2 accepted

0 votes
Answer accepted
chitra_amarnath_veriforce_com November 18, 2018

We use Zephyr for our tests management and it has been greatly helpful to manage our testing cycles not only for the releases but also for sprint cycles. We have distributed teams across different time zones and with Zephyr we have been able to collaborate around the clock successfully. With their latest version, organizing the test cases in folders is being very helpful as I am able to group then at epic level and assign to respective team members based on their domain knowledge. I was also able to report overall testing progress with % complete, passed, failed & blocked. I could also clone cycles, export test cases & import them into next cycles etc. I also could save test execution filters, add them to the main dashboard using the widgets for reporting. 

I haven’t used personally Xray, but so far we are very satisfied with Zephyr and have been using it for the past two plus years after moving from TFS.

 

Thanks,

Chitra 

priya abuj November 20, 2018

Hi Chitra

Are you using Zypher also for reporting purpose?

I mean sending some reports[as email attachment/links for Jira Report] to client about Defects status or Requirement coverage per Sprint?

 

Thank you 

Priya

chitra_amarnath_veriforce_com November 20, 2018

Hi Priya,

Not so much yet as we have some custom reporting format. I usually export the defects using custom queries for our management team, who also view the sprint board on a daily basis to see the real time stats via dashboards.

Traceability report is popular one that I use a lot. Our customer base do not access Jira directly.

Thanks,

Chitra

0 votes
Answer accepted
kg4ysy November 15, 2018

We use Zephyr at our office and it has been very beneficial. We build tests that we can reuse. Each time we have a new release, we start a new test cycle for that particular version. JIRA integrates well with Zephyr in that manner. The test cycles contain tests. Tests contain test steps. Each test step and test as a whole is tracked with its current status (WIP, unexecuted, pass, fail, etc.) One of my favorite features is being able to open a new bug from the test execution screen that automatically links to that particular test execution. Each test execution is also linked on the test itself. This allows us to see the success of the test over time. Each defect we find is linked as well, which allows us to see where the pain points are. There are some dashboard widgets too. We don't use those since we are manage a lot of small projects, but they provide a lot of useful elements for the current testing environment. I have no experience with Xray, but I'm happy with Zephyr.

4 votes
Mark Bentsen November 8, 2019

We have Xray. Hope this helps.

Test-Mgmt.gifIf any data in the image is incorrect, please comment and provide source material so I can update this for the community.

Disha Thakkar _QMetry_
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
November 12, 2019

This is good consolidated comparison between different test management apps.

We would like to highlight few points here which are present in QMetry but mentioned “as not present” in this comparison –

  1. QMetry allows organizing test cases and other artefact in hierarchical folder structure for easy navigation and consolidation of testing artefact.
  2. QMetry provides “Automation Attributes” that are used to identify Automated Test Cases uniquely.
  3. It can create test cases from automated test results and user can also get a summarization of the imported results.
  4. As mentioned by you, it also supports automation frameworks like QAS and HP UFT along with Cucumber, TestNG, Junit, SpecFlow.
  5. Customization of test step execution is possible through adding, editing, deleting and prioritizing test execution status.

To get more information, you can mail us on info@qmetry.com. You can also sign up for QTM4J free trial on Atlassian Marketplace.

We are curious to know how did you evaluate all these apps? This can help users of Jira community to understand how you analysed different apps to get this comparison.

From product owner’s perspective, this will help us understand the user’s needs and feedback and ultimately can provide the best in market solution catering to their problems.

Mark Bentsen November 13, 2019

I made some updates in the chart. You will see those above. Regarding your question "...We are curious to know how did you evaluate all these apps? "

I did review the websites of the companies to see what they were saying about each other. This was the data I put the least weight in. As time went by, there are several areas where this is incorrect. For example, Xray says Zephyr does not support BDD. Well... Smartbear (Zephyr) purchased Cucumber.

Keep in mind, we already license Xray & Quality Center, I just wanted to determine if there were better options available. QC and qTest are stand alone solutions. The ones above are lighter weight, and integrate directly to Jira.

Then I reviewed all the comments on the marketplace to see where users were specifically unhappy with aspects of the tool. From this collection I researched these specifics on the web and in release notes to see if these items were resolved or if they persist.

Then I posted an initial version of this image to see if vendors would correct me (just as you did in this comment - Thank You!) or members of the community for that matter. Sure enough, they did. I made updates, and that is what you are seeing above. The only Yes across the board, is the REST API. Zephyr was a 'no' for this, but I was informed they created the Zephyr REST API (ZAPI).

Realize that the number of installs and comments changes. It is current as of the time of this post. If you want to see which products are increasing or declining, tracking this rate over time should provide some insights.

Like # people like this
Brian Murray March 16, 2020

Hi @Mark Bentsen 

We are considering XRay for Jira as a replacement for our existing test Case management system.  However, we are concerned about the impact this might have on a regular (non-tester) Jira users.  I work at Red Hat and we have a heavy load of continuous integration tests running which results.  I was wondering if you could she some more light on your experience with XRay either here or via email.

thanks,

Brian M. (red hat)

Mark Bentsen March 16, 2020

What are the impacts you are thinking of? It is important to understand that XRAY is integrated to Jira. You will have new issue types and additional left hand navigation options. It should not impact your Jira instance so long as your workflows are not overly complex. They provide docs on this in performance and reliability section. If you have more specific questions, I'm happy to share from my experience.

Brian Murray March 16, 2020

Thanks Mark,  So we have done some initial testing on a single server environment as well as a 3 node cluster.  I have reviewed most of the docs and we have been working closely with both the XPand IT Xray folks and the Atlassian people.  Our existing TCMS workflows are not that complex.  I suspect our Jira workflows are probably more complex than not.  We are considering dropping out existing TCMS in favor of using Jira because our developers and PM people already use that. 

The hope is that everyone's lives will be simpler if everything is in one place.  Our concern is that we upload 20 millions test results every month to our current system.  it took us a lot of tweaking and work to get the current system robust enough to handle that kind of load.  We isolate the Uploading of automated results to 3 nodes while the UI users are isolated on another 2 nodes.  So our number 1 concern is that the automated uploading of results that goes on 24/7 not affect the UI performance.

thanks for any help,

Brian

Mark Bentsen May 5, 2020

So... Brian, 20 million test results a month is outside my experience. Hopefully you do not have many 'flaky' tests in that mix. That would be a lot of potential false positives to comb through. I'd maximize the trial time to see if XRAY can handle that load. They have been getting better and better at improving performance (reducing latency). We have some robust workflows. They recommend to keep them simple. So far, so good in the current version. Again... 20 million - whoa. What are you doing automated checking for? Could you achieve that desired coverage with a significant subset? How do you care for, feed, and groom such a large library? I would also add, we recently added Xporter. I really like it. Others in my org are not so wild about the fact it takes up prominent real estate on the Jira issue page.

Brian Murray May 5, 2020

Thanks Mark,

Most of that 20 mil is continuous integration stuff that is fed in from Jenkins.  I am in charge of the TCMS, however I try not to meddle into the tasks of the teams that are responsible for the actual testing.  I provide guidelines, etc.  As far as reducing the number of tests that is not really feasible.  We have gotten lots of pushback from the teams that they are only doing what they need to do and that if the TCMS system can't handle it then they feel like we should just choose one  that can handle it.

 

Brian

Mark Bentsen May 12, 2020

So, after additional consideration, I would position myself by saying "Most modern solutions are going to be fine for your needs." Lots of purchase making people in the enterprise are hoping to find the 'one right tool' for their testing needs. Salespeople for vendors are helping to sell this perspective. This is what I believe... It is more like getting married. Is there only one person who will be your ideal mate for life? mmm... not likely. Much of it is about you, and compatibility, and being up front, and other important things I can't come up with right now. If you have a sufficient tool, good processes, and obey the rules of the road; chances are it will go well for you when you purchase the solution. XRAY, Zephyr, QMetry, etc. If you are wheels off, the tool is not going to save you. 

 

Also, please try to divorce Excel for test management, and beware of freeware (there is no such thing. Time and attention are expensive non-replenishable resources). 

Like # people like this
Arek Kachniarz September 15, 2020

beautiful summary! thanks!

2 votes
Brian Walker November 10, 2019

Hi Priya,

My vote goes to QMetry Test Management for Jira and I am using it personally from the past couple of years.

Here is the Marketplace Link:

https://marketplace.atlassian.com/apps/1215144/qmetry-test-management-for-jira?hosting=cloud&tab=overview

- Brian

0 votes
Jayesh Wankhade October 13, 2020

You can read my article here. It helps you to decide what to select as per your requirement. I am hoping it will solve your problem.

https://community.atlassian.com/t5/Jira-articles/Comparison-between-Zephyr-amp-Xray/ba-p/1504907

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events