There are a host of plugins for test management. I reviewed five of these. My results are below. It has been challenging getting accurate data. Much of the info out there appears to be created by the vendor's marketing teams.
Let me know if you disagree with anything here or if there are key criteria that is omitted. If all five plugins had the same feature or functionality, I did not include it in my list. You will just see the deltas. Below the table I explain my method for putting this together.
My method:
I did review the websites of the companies to see what they were saying about each other. This was the data I put the least weight in. As time went by, there are several areas where this is incorrect. For example, Xray says Zephyr does not support BDD. Well... Smartbear (Zephyr) purchased Cucumber.
Keep in mind, we already license Xray & Quality Center, I just wanted to determine if there were better options available. QC and qTest are stand alone solutions. The ones above are lighter weight, and integrate directly to Jira.
Then I reviewed all the comments on the marketplace to see where users were specifically unhappy with aspects of the tool. From this collection I researched these specifics on the web and in release notes to see if these items were resolved or if they persist.
Then I posted an initial version of this image to see if vendors would correct me or members of the community for that matter. Sure enough, they did. I made updates, and that is what you are seeing above. The only Yes across the board, is the REST API. Zephyr was a 'no' for this, but I was informed they created the Zephyr REST API (ZAPI).
Realize that the number of installs and comments changes. It is current as of the time of this post. If you want to see which products are increasing or declining, tracking this rate over time should provide some insights.
These were some of my conclusions:
Please share your experiences and I invite you to correct anything I got wrong. Hope this helps the community, especially those who are serious about testing and quality!
Thanks for commenting Marcin. I'll answer the bullets in order:
Bullet 1 - Based on comments from the Marketplace for the plugin here.
Bullet 2 - I will update my list. Thanks for clarifying this.
Bullet 3 - I used info here. Do you believe TesFLO has functionality as compared there? I'm inclined to agree with what you have included, but would like more details. I'm guessing that TestFLO creates 'shells' of test cases based on the templates.
Bullet 4- From this comment on Marketplace: "I really like the product, I think it's certainly one of the best in its kind. Two major things that I really want changed though are:
- When creating a test plan and adding test case templates to the plan - there needs to be a way to somehow filter through all of the existing templates. Handpicking manually some 50-70 templates out of the pool of 3000 is just not realistic. Maybe get all templates from a certain pre-saved filter, or something like that?
- I'd prefer for test cases to be linked to templates. They are now, but not in the best way - there's a back link from the case to the template but what I'd like to see is all test cases originated from any given template along with their latest execution status. I believe it would greatly help with overall testing visibility."
It was followed up with the following and therefore may be mitigated: "...Thank you very much for sharing your opinion with us! Its seems that you speak on behalf of other SuiTest's users ;) that requires more advanced test case templates picker as well. We released a powerful JQL queries support in v2.1 Second feature is already there :). Please, check Use Case #202: Show all Test Cases created from given Test Case Template"
Mark,
Thank you for your reply.
You could also try to contact each support team directly to experience the service by yourself.
Yes, you can create these empty "dummy" Test Cases to update them with test results, so in fact, you don't have to import all test results, but only the ones added to the Test Plan as Test Cases.
The comment you've mentioned is more than six years old. Since then, we have released countless versions. Both problems mentioned in the comment were resolved a long time ago.
In general, comments from the Atlassian Marketplace will not give you a true overview of the add-on. You should take into consideration to trial and evaluate them directly.
Kind regards,
Marcin, TestFLO Team
Thanks @Mark Bentsen
Honestly, I think each product brings its value to the table but of course there are differences and pros and cons.
Xray, for example, gives power back to the user so she/he can fine-tune their own process to the needs and how the team wants to work. It is highly coupled with Jira concepts which allows you to take advantage of many things out-of-the-box, not just for test cases but also for test plans, test executions, etc.
And as many teams have different processes and also different tools/apps to cover different needs, Xray lets users use all of them in the best way possible.
Thus, you have many integrations with all the Atlassian marketspace and more (e.g. TeamCity).
Regards
Sergio
It is really a GOOD matrix!
Seems you missed another good one named synapseRT?
It has almost all features listed in the table, and it connects Requirements / Test Case / Test Results / Defects very well. Hope you can check this product and add it into your list as well:).
Regards,
Cheney
Thanks Cheney. I will look into it in more detail when I do another round of analysis.
From an early look it appears that the users are very positive about the tool since the 8.0 release. It is probably the best choice if you have some portion of your team in China. Also, if your testware organization lends itself to being stored in a tree structure, this is a strength. Thanks for adding it as an option. Right now there appear to be 602 installs and 49 comments on the Marketplace. The rating is close to 3.5 from the user community.
Nice comparison. We tried a few of the plugins but found they were bloated and had too much unwanted functionality we never used. We therefore looked for a plugin that was built specifically for BDD and have used AssertThat.The support is really good and they have added a few small features which were causing us some pain.
The plugin gives us all we need for our BDD setup including:
- linking user stories and features/scenarios
- integration with our test automation framework
- uploads test automation results.
- traceability
Cheers
EK