Comparison of TestFLO vs XRay vs Zephyr on JIRA Server

[Benjamin Weinheimer-Erben (mgm-tp) · Accepted solution]

Hi,

at least for TestFLO and Zephyr you will find some good information here: http://blog.mgm-tp.com/2016/07/test-management-with-jira/

greetings

Benjamin

[kouros goodarzi]

Yes thank you, I have seen that. I was very much interested in knowing if someone has tried XRay. Looking at the objects in JIRA it seems the most complete of them all.

[Xpand IT]

Hi @kouros goodarzi,

thanks! Because Xray uses JIRA issues it brings you a lot of additional features implicitly ootb, such as customization of screens, workflows, additional "attributes" (i.e. custom fields), ability to include them directly in Agile boards, accountability, changes tracking, among many other.

A great thing, in our opinion, is that you can do things such as supporting multiple project organization scenarios, which can be quite useful for many organizations.

Regards,

Xray for JIRA Team

[Attila Zs]

@Xpand ITthe correct link for Project organization

https://confluence.xpand-addons.com/display/XRAY/Project+Organization

[Xpand IT]

Hi,

You may find a comparison between Xray and Zephyr here, and we are able to provide you a more exhaustive comparison if you need.

There are some more available if you google for it, such as this one.

Tools evolve with time, so some comparisons may be outdated though and it all will depend in what you're looking for.

We do believe that Xray is a great tool for management of both manual and automated tests, either Cucumber or any other. If you want to give a look, we have extensive documentation for automated tests including tutorials. And we believe it's the best, not because we say so, but because our customers say so. 

And if you're keen to Agile and have on real-time feedback, because we use JIRA issues you can take many benefits of it such as having the ability to include them in your boards.

Also, whenever you look at a tool you should think in several aspects, such as the time in the market (we are more than 3 years), the quality of the support you get (check reviews from our customers),  the quality of the documentation and other goodies such as regular webinars, video tutorials. 

Please don't hesitate to reach us through our service desk, and we will be glad to help you in your needs and even provide a demo or more resources if you need.

Best regards,

Xray for JIRA Team

[Deleted user]

https://www.getxray.app/xray-vs-zephyr/?gclid=EAIaIQobChMI9_TckZOX5AIVUFmGCh2FIAwUEAAYASAAEgL_fvD_BwE

[Mark Bentsen]

If any data in the image is incorrect, please comment and provide source material so I can update this for the community.

This image is a comparative analysis based on documentation on the web. Part of my analysis was based on going through the reviews from users on the Marketplace so as to not only use the marketing materials from the vendors.

We use Xray, have had a long history with QC and recently conducted a pilot of qTest (QC and qTest are 'stand alone' and are not integrated directly with Jira). Several members of our team really like Xray. Limitations I see are its reports and that you must purchase Xporter to bulk export tests.

I'm interested in the feedback from others regarding these conclusions for the following test management solutions:

• Zephyr – Easy to implement, but limited customization possibilities. Poor customer service. Customizable. Popular. Smartbear product a plus if you use Ready! API. Extensive REST API.
• TM4J (Adaptavist) – very positive support. Use of REST API to import automated test results. Easy to import test cases.
• TestFLO – Automation plugin at additional cost. Lots of configuration, however then you are locked into those choices. (i.e. Handpicking manually some 50-70 templates out of the pool of 3000 is just not realistic.) Changing template is challenging. Limited JQL features.
• Qmetry – Better for manual tests. Good at reuse of test cases. Variety of reports. Firefox issues. 

[Aron Gombas _Midori_]

Related to your note about bulk exporting tests, I wanted to mention that the Better PDF Exporter and Better Excel Exporter apps can export Zephyr tests, execution and test results to PDF and Excel, respectively. These can solve many Zephyr exporting/reporting use cases due to their rich feature set and flexibility.

Disclaimer: these are two paid apps and I am working on these.

[Mark Bentsen]

Thanks for providing these options Aron. I had not previously heard about these. They will fill in an important gap for the Zephyr product.

[kg4ysy]

The links provided by others are good. We use Zephyr for JIRA for our QA Team. So far it has been a big success. Zephyr uses JIRA issues to manage the tests. The test cycles are managed from within JIRA which keeps us from having to move between software. That was a huge pain point in the past. The integration with project versions and settings also saves us a lot of time. We can tie our tests to projects and versions so we can see our progress over time. I don't have any experience with other testing plugins in JIRA, but we gave Zephyr a try and stuck with it. I'd recommend it.

[Aron Gombas _Midori_]

I think that there are even more candidates to be included in such comparison, like Kanoah Tests. I myself would also be very interested in this.