zephyr, Enterprise tester, SynapseRT - user experiences

[meena radhakrishnan]

I am in the process of evaluating a suitable test management tool that can be plugged into JIRA and would like to know user experiences if anyone has used any of these tools.

Also if there is any other tool besides these please list them here as well.

thanks

[JohnA · Accepted solution]

Have you checked out the user reviews for these plugins on the Atlassian Marketplace? For example, check out the reviews for SynapseRT:

https://marketplace.atlassian.com/plugins/com.go2group.jira.plugin.synapse

[Ellen Feaheny [AppFusions]]

Well - a few key things that stick out to me with Enterprise Tester, I think are:

• VERY strong integration with JIRA and Confluence (for reporting) - and constant release feature updates schedule, it seems.
• TQL - their testing query language which is super powerful.
• Extensible with their own API - so you can extend more to your testing requirements envrionment.
• Good customer "partner".. responsive with iterations.

Try it yourself - don't take my word!

[MatthewB]

I have no direct experience with either SynapseRT or Enterprise Tester, but we evaluated Zephyr on my last project and we had a pretty good experience with it. There were some things I felt it could do better, but they were mostly just productivity improvements. If you need to maintain a base of manual tests, or instructions on how to run semi-automated tests, it will definitely work for you. Use something like Bonfire or Jing to capture your exploratory testing runs, then play them back and document them in a tool like this to capture the essence of what you want to be able to verify in the future.

A little context on what we used this for. We already had a pretty solid set of fully automated tests, but we also had a good number of automation-assisted tests that needed to be run and verified by real people, and a handful of fully manual tests (think inverted testing triangle). We used Zephyr to track the execution of everything that wasn't fully automated, and against what "Build" of the software we had run it against. We had set up our build pipeline so that the same code would result in binaries with the exactly same MD5 hash, so those were our "Builds". That way we didn't waste time running tests against code we had already run them against.

Before getting into the nitty gritty I want to say what I don't believe this is. This isn't a QTP-esque system and I don't think they mean it to be. It is only meant to track progress of people running tests and link those tests to other issues in Jira (bugs, features, stories, etc.). This aids in various traceability exercises (e.g. this test broke so what requirements are now invalidated), and acts as a basic store of test knowledge. This as opposed to something like SynapseRT which tries to be both Requirements management/traceability and test case management.

Some advice to make sure you get the most of out it:

1. Be sure to fill out the build field on your Test Cycles so you know what tests you've covered and which you haven't. If you can get better defined configuration management (e.g. for individual components, etc.) I suggest breaking out test cycles for each of those. That way you don't waste time testing something that hasn't changed.
2. Make sure you clone your test cycles for each new build. Since this will likely be overwhelming at first, just pick a "good" build for some time period and run your tests against that.
3. Don't have tests that aren't linked to requirements/stories. If you can't trace them then why is there a test for it? You'll also be happy you have all that data at your fingertips when you start to build your test library.
4. Stick with it, the system is only as good as the data you put into it. If you don't track when you're running tests and what those tests are trying to verify you're not going to get all the value out of it.

What it can do better:

1. Mass cloning of Test Cycles. This was a huge annoyance when moving from one build to another while staying within the same Jira version. I just wanted to bring everything to a new build and move on, but had to clone them all individually.
2. Automatic cloning of test cycles triggered by some integration with a CI system.
3. The ability to share tests between projects. If you have a lot of projects with similar/identical requirements like we did (non-functional requirements) you will wish that you could track and disseminate the best practices for verifying those requirements to members of other teams. You can, but not using this system, and you can't easily share the tests between projects.
4. OnDemand support. This wasn't a problem for us since we ran a hosted instance but I imagine it will be for some.

The bottom line. It's a tool, not a panacea. You need to have good all around practices and track diligently in order to get the most out of this, but I imagine that is true for any system. If you just need a test case management solution for Jira, then Zephyr should suit your needs. If you want requirements traceability in addition to test case management then SynapseRT may be a better fit.

[SteveR]

Matthew, for your manual tests, do you put the test scripts (each step, expected results,etc.) in Zephyr?

[MatthewB]

@Steve - Yes we did. For manual or automation-assisted tests we would put the actual stepts and expected results in Zephyr, and then scripts that needed to be run (e.g. Python, etc.) we stored alongside our codebase in source control to keep good configuration management.

[SteveR]

We have used SynapseRT for a couple of months. It runs inside of JIRA, but is missing functionality that is fundamental to test management. For example, there's no way to create a test case with discreet steps, with each step having a desc, expected result and actual result. We're hoping for improvements in the near future.