It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Comparison of TestFLO vs XRay vs Zephyr on JIRA Server

 

5 answers

1 accepted

Hi,

at least for TestFLO and Zephyr you will find some good information here: http://blog.mgm-tp.com/2016/07/test-management-with-jira/

greetings

Benjamin

Yes thank you, I have seen that. I was very much interested in knowing if someone has tried XRay. Looking at the objects in JIRA it seems the most complete of them all.

Hi @kouros goodarzi,

thanks! Because Xray uses JIRA issues it brings you a lot of additional features implicitly ootb, such as customization of screens, workflows, additional "attributes" (i.e. custom fields), ability to include them directly in Agile boards, accountability, changes tracking, among many other.

A great thing, in our opinion, is that you can do things such as supporting multiple project organization scenarios, which can be quite useful for many organizations.

Regards,

Xray for JIRA Team

Like ken_tannenbaum likes this

Hi,

You may find a comparison between Xray and Zephyr here, and we are able to provide you a more exhaustive comparison if you need.

There are some more available if you google for it, such as this one.

Tools evolve with time, so some comparisons may be outdated though and it all will depend in what you're looking for.

We do believe that Xray is a great tool for management of both manual and automated tests, either Cucumber or any other. If you want to give a look, we have extensive documentation for automated tests including tutorials. And we believe it's the best, not because we say so, but because our customers say so. 

And if you're keen to Agile and have on real-time feedback, because we use JIRA issues you can take many benefits of it such as having the ability to include them in your boards.

Also, whenever you look at a tool you should think in several aspects, such as the time in the market (we are more than 3 years), the quality of the support you get (check reviews from our customers),  the quality of the documentation and other goodies such as regular webinars, video tutorials

Please don't hesitate to reach us through our service desk, and we will be glad to help you in your needs and even provide a demo or more resources if you need.

Best regards,

Xray for JIRA Team

If any data in the image is incorrect, please comment and provide source material so I can update this for the community.

Test-Mgmt.gif

This image is a comparative analysis based on documentation on the web. Part of my analysis was based on going through the reviews from users on the Marketplace so as to not only use the marketing materials from the vendors.

We use Xray, have had a long history with QC and recently conducted a pilot of qTest (QC and qTest are 'stand alone' and are not integrated directly with Jira). Several members of our team really like Xray. Limitations I see are its reports and that you must purchase Xporter to bulk export tests.

I'm interested in the feedback from others regarding these conclusions for the following test management solutions:

  • Zephyr – Easy to implement, but limited customization possibilities. Poor customer service. Customizable. Popular. Smartbear product a plus if you use Ready! API. Extensive REST API.
  • TM4J (Adaptavist) – very positive support. Use of REST API to import automated test results. Easy to import test cases.
  • TestFLO – Automation plugin at additional cost. Lots of configuration, however then you are locked into those choices. (i.e. Handpicking manually some 50-70 templates out of the pool of 3000 is just not realistic.) Changing template is challenging. Limited JQL features.
  • Qmetry – Better for manual tests. Good at reuse of test cases. Variety of reports. Firefox issues. 

Related to your note about bulk exporting tests, I wanted to mention that the Better PDF Exporter and Better Excel Exporter apps can export Zephyr tests, execution and test results to PDF and Excel, respectively. These can solve many Zephyr exporting/reporting use cases due to their rich feature set and flexibility.

Disclaimer: these are two paid apps and I am working on these.

jira-zephyr-test-executions

Like Mark Bentsen likes this

Thanks for providing these options Aron. I had not previously heard about these. They will fill in an important gap for the Zephyr product.

The links provided by others are good. We use Zephyr for JIRA for our QA Team. So far it has been a big success. Zephyr uses JIRA issues to manage the tests. The test cycles are managed from within JIRA which keeps us from having to move between software. That was a huge pain point in the past. The integration with project versions and settings also saves us a lot of time. We can tie our tests to projects and versions so we can see our progress over time. I don't have any experience with other testing plugins in JIRA, but we gave Zephyr a try and stuck with it. I'd recommend it.

0 votes

I think that there are even more candidates to be included in such comparison, like Kanoah Tests. I myself would also be very interested in this.

Suggest an answer

Log in or Sign up to answer
TAGS

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you