nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Not all test management plugins are created equal

Mark Bentsen

There are a host of plugins for test management. I reviewed five of these. My results are below. It has been challenging getting accurate data. Much of the info out there appears to be created by the vendor's marketing teams.

Let me know if you disagree with anything here or if there are key criteria that is omitted. If all five plugins had the same feature or functionality, I did not include it in my list. You will just see the deltas. Below the table I explain my method for putting this together.

My method:

I did review the websites of the companies to see what they were saying about each other. This was the data I put the least weight in. As time went by, there are several areas where this is incorrect. For example, Xray says Zephyr does not support BDD. Well... Smartbear (Zephyr) purchased Cucumber.

Keep in mind, we already license Xray & Quality Center, I just wanted to determine if there were better options available. QC and qTest are stand alone solutions. The ones above are lighter weight, and integrate directly to Jira.

Then I reviewed all the comments on the marketplace to see where users were specifically unhappy with aspects of the tool. From this collection I researched these specifics on the web and in release notes to see if these items were resolved or if they persist.

Then I posted an initial version of this image to see if vendors would correct me or members of the community for that matter. Sure enough, they did. I made updates, and that is what you are seeing above. The only Yes across the board, is the REST API. Zephyr was a 'no' for this, but I was informed they created the Zephyr REST API (ZAPI).

Realize that the number of installs and comments changes. It is current as of the time of this post. If you want to see which products are increasing or declining, tracking this rate over time should provide some insights.

These were some of my conclusions:

Zephyr – Easy to implement, but limited customization possibilities. Poor customer service. Customizable. Popular. Smartbear product. Extensive REST API.
TM4J (Adaptavist) – very positive support. Use of REST API to import automated test results. Easy to import test cases.
TestFLO – Automation plugin at additional cost. Most configurable choice, however then you are locked into those choices. (i.e. Handpicking manually some 50-70 templates out of the pool of 3000 is just not realistic.) Change is challenging. Limited JQL features.
Qmetry – Better for manual tests. Good at reuse of test cases. Variety of reports. Firefox issues reported.
Xray – If you have custom workflows, this is the best choice. Reporting could be improved. Also, would like to be able to bulk export test cases with steps without having to purchase Xporter. Overall, this is my recommendation if I was spending my own money. Recommend using a blend of tests that are in there own Jira project (Regression) and inside existing projects (Functional). I like the ability to import bugs and test cases.

Please share your experiences and I invite you to correct anything I got wrong. Hope this helps the community, especially those who are serious about testing and quality!

Comments

Marcin Żurawiecki

Hello @Mark Bentsen!

I am the product owner of TestFLO and I have to admit that you did a really great job here! Thank you for including our add-on in your review. We appreciate the feedback. However, I'd like to discuss and ask about a few things:

How did you evaluate the support services of each add-on?
The thing about the "Version/History of Test Cases". The whole change history of a Test Case is stored on the issue in the "History" tab. What's more, you can use the snapshots functionality to save the state (like a version) of a test when making a workflow transition.
I am not sure if I understand the point "Create Test Cases from automated tests results" right, but TestFLO actually creates the Test Cases based on the results imported from the CI, what's more you can configure the add-on to automatically create Test Case Templates to create "dummy" Test Cases in Test Plans update them when running your automated tests.
You've mentioned about not realistic handpicking templates in your conclusion. Test Case Templates can be searched through using JQL, saved filters, projects or simple text. What's more, you can use the tree structure to find the tests and create new Test Plans or add them to the existing ones. Could you describe the problem in more detail?

Nevertheless, the comparison is really valuable and makes me realize that there are still areas that we need to work on. Once again - thank you!

Kind regards,
Marcin, TestFLO Team

Sergio Freire - Xblend

Thanks @Mark Bentsen

Honestly, I think each product brings its value to the table but of course there are differences and pros and cons.

Xray, for example, gives power back to the user so she/he can fine-tune their own process to the needs and how the team wants to work. It is highly coupled with Jira concepts which allows you to take advantage of many things out-of-the-box, not just for test cases but also for test plans, test executions, etc.

And as many teams have different processes and also different tools/apps to cover different needs, Xray lets users use all of them in the best way possible.

Thus, you have many integrations with all the Atlassian marketspace and more (e.g. TeamCity).

Regards

Sergio

Cheney Ma [Go2Group]

Hi @Mark Bentsen

It is really a GOOD matrix!

Seems you missed another good one named synapseRT?

It has almost all features listed in the table, and it connects Requirements / Test Case / Test Results / Defects very well. Hope you can check this product and add it into your list as well:).

Regards,

Cheney

Mark Bentsen

Thanks for commenting Marcin. I'll answer the bullets in order:

Bullet 1 - Based on comments from the Marketplace for the plugin here.

Bullet 2 - I will update my list. Thanks for clarifying this.

Bullet 3 - I used info here. Do you believe TesFLO has functionality as compared there? I'm inclined to agree with what you have included, but would like more details. I'm guessing that TestFLO creates 'shells' of test cases based on the templates.

Bullet 4- From this comment on Marketplace: "I really like the product, I think it's certainly one of the best in its kind. Two major things that I really want changed though are:
- When creating a test plan and adding test case templates to the plan - there needs to be a way to somehow filter through all of the existing templates. Handpicking manually some 50-70 templates out of the pool of 3000 is just not realistic. Maybe get all templates from a certain pre-saved filter, or something like that?
- I'd prefer for test cases to be linked to templates. They are now, but not in the best way - there's a back link from the case to the template but what I'd like to see is all test cases originated from any given template along with their latest execution status. I believe it would greatly help with overall testing visibility."

It was followed up with the following and therefore may be mitigated: "...Thank you very much for sharing your opinion with us! Its seems that you speak on behalf of other SuiTest's users that requires more advanced test case templates picker as well. We released a powerful JQL queries support in v2.1 Second feature is already there . Please, check Use Case #202: Show all Test Cases created from given Test Case Template"

Mark Bentsen

Thanks Cheney. I will look into it in more detail when I do another round of analysis.

From an early look it appears that the users are very positive about the tool since the 8.0 release. It is probably the best choice if you have some portion of your team in China. Also, if your testware organization lends itself to being stored in a tree structure, this is a strength. Thanks for adding it as an option. Right now there appear to be 602 installs and 49 comments on the Marketplace. The rating is close to 3.5 from the user community.

unknown

Nice comparison. We tried a few of the plugins but found they were bloated and had too much unwanted functionality we never used. We therefore looked for a plugin that was built specifically for BDD and have used AssertThat.The support is really good and they have added a few small features which were causing us some pain.

The plugin gives us all we need for our BDD setup including:

- linking user stories and features/scenarios
- integration with our test automation framework
- uploads test automation results.
- traceability

Cheers
EK

Marcin Żurawiecki

Mark,

Thank you for your reply.

You could also try to contact each support team directly to experience the service by yourself.

Yes, you can create these empty "dummy" Test Cases to update them with test results, so in fact, you don't have to import all test results, but only the ones added to the Test Plan as Test Cases.

The comment you've mentioned is more than six years old. Since then, we have released countless versions. Both problems mentioned in the comment were resolved a long time ago.

In general, comments from the Atlassian Marketplace will not give you a true overview of the add-on. You should take into consideration to trial and evaluate them directly.

Kind regards,
Marcin, TestFLO Team

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

atlassian-marketplace

xray

zephyr

test-management

test

tm4j

quality-assurance

qmetry

testflo