How do I execute automated tests from JIRA? [Zephyr]

Sorry to be slow...

We've just been playing with Zephyr with the idea that an automated stack could be set up as follows:

  • Zephyr plug-in for JIRA (write test cases linked to requirements)
  • Zephyr community or enterprise edition (automatically sync with Zephyr plug-in Tests, then further details added such as link to ZBOTs and scripts)
  • ZBOT deployed on to system under test
  • C# running script on system under test

We were hoping the following:

  • The 'EXECUTE' button in the Zephyr plug-in would actually 'execute' a Test through the above stack and render the result directly back to the Zephyr plug-in within JIRA.
  • This would allow:
  1. BAs to create test cases as part of their New Feature/Stories elaboration
  2. Direct linkage of New Features/Stories to Test issues (the 'Test' being the Zephyr plug-in Issue type)
  3. Direct linkage of Test results through to JIRA Defects

I feel like I lost the plot somewhere as the 'EXECUTE' button in Zephyr plug-in only seems to be a manual field. Though the above objective seems intuitively reasonable!

Questions...

a) Can Zephyr achieve what we'd like to do? (i.e. direct execution of automated tests from JIRA and result passed back to JIRA)

b) If not, why not... it seems a reasonable objective?

c) Is there another product which would allow this?

Many thanks for any advice.

7 answers

1 accepted

Hi Simon,

Zephyr here. It sounds like you're merging the performance of two of our products in your mind. Let me explain.

Questions...

a) Can Zephyr achieve what we'd like to do? (i.e. direct execution of automated tests from JIRA and result passed back to JIRA)

Not yet.

Currently the Zephyr Enterprise Edition/Community Edition are unique products that don't interact with and are sold seperatly from the Zephyr for JIRA plugin. (Note they do come with a Zephyr Enterprise Connector Plugin, which offers different functionality than the ZFJ plugin.)

These two products do have the Zbot you are speaking of. When tests are executed from ZEE or ZCE, the results come back into ZEE or ZCE. Both of these products offer great integration with JIRA. For example, on the Zephyr Requirement side, you can see current content of any user story or issue you "import" (where we import only pointers to the current content) from JIRA. As well, you can create, update and edit defects or other issues from the Zephyr Defect Tracking UI which are communicated directly to the JIRA database. So there is no syncronization going on at all. As well, ZEE and ZCE integrate with both JIRA On-Demand and JIRA On-Premise Deployments.

As well, ZEE and ZCE come with APIs which can be perfect for CI scenarios. Many of our customers do use the ZEE APIs for CI.

The Zephyr for JIRA plugin doesn't yet offer APIs as a publically supported feature.

b) If not, why not... it seems a reasonable objective?

Yes it does. APIs for Zephyr for JIRA are on our roadmap and we'll be excited to offer them. This is already a highly requested feature.

Follow our Blog for the any and all current announcements about our product: http://blogs.yourzephyr.com/

Kind Regards,

Chris

Thanks Chris.

Yep - we got there today & realised that the 2 products (Zephyr plug-in & ZEE/ZCE) didn't join up. Our specific challenges are:

  • We want our BAs to start to generate requirements and associated tests. For this reason it would add to complexity (and act as a barrier) if we presented them with 2 different looks for requirements & tests => there is a strong incentive to keep both in JIRA and therefore use Zephyr plug-in rather than ZEE/ZCE.
  • We now understand it is possible to use the Zephyr plug-in along with the JIRA REST APIs & Bamboo to construct a more hybrid CI/auto-test model but I guess it has the following limitations..?:
  1. 'Execute' in the Zephyr plug-in will still not trigger anything to run? i.e. there is no way of generating an external command from the Zephyr plug-in?
  2. We can trigger execution from outside JIRA and then use the JIRA API to update the PASS/FAIL on individual tests? [Note we would want to use the test cycle functionality to differentiate between regression test runs etc.] This would avoid manually changing the status of all tests after a run.
  3. It would need some [much?] work to construct this stack..?

Any further comments much appreciated.

Thanks for your advice.

Simon

Hi Everyone,

Just to update this thread one last time, ZAPI is now available. ZAPI adds API capabilities to Zephyr for JIRA. You can find the ZAPI Marketplace page by following the link below.

https://marketplace.atlassian.com/plugins/com.thed.zephyr.zapi

Regards,

Daniel

Two years later... is Simon's use case possible today (using Zephyr)?

  1. BAs to create test cases as part of their New Feature/Stories elaboration
  2. Direct linkage of New Features/Stories to Test issues (the 'Test' being the Zephyr plug-in Issue type)
  3. Direct linkage of Test results through to JIRA (eg. failed/passed)

Hi Bart, Yes, we now have the ZAPI plugin. This plugin is purchased alongside Zephyr for JIRA Server and gives it a full suite of powerful REST API. You can use JIRA's API to create a Test issue and then ZAPI to create the steps in that issue. You can issue link Tests to another other issue type You can also use the ZAPI REST API to bring in automation results directly into Zephyr for JIRA For more information on ZAPI, please see full details here: https://marketplace.atlassian.com/plugins/com.thed.zephyr.zapi/server/overview Regards, Daniel

I've now watched a few videos of Enterprise Tester to see if it gives us the missing automation directly from JIRA. It doesn't seem to...

  • It can bi-directionally synchronise on selected Issue types (and Issue fields) from JIRA: therefore a bespoke Test* Issue type in JIRA could be used as an import type to E.T.
  • Execution would be in Enterprise Tester
  • The bi-directional update suggests results could be populated back into JIRA Tests* - and our workflow managed through JIRA Issue types Stories/Tests*/Defects
  • (*not a native type)

So the advertised 'integration' with JIRA seems more about data exchange than runtime execution - for both tools. As both tools also have requirement and defect types I'm beginning to wonder why bother with JIRA at all... i.e. why not use one of these to manage the entire workflow?

Hi Simon

Have you consider using Behave for JIRA?

All Accetance Tests related to JIRA issues are downloaded from JIRA using custom build tools and then you can use any Gherkin based Tests tool for execution. For C# this would be SpecFlow would be the Gherkin based tool of choice.

Test results wouldn't be displayed in JIRA but through your CI system of choice.

Alan

Thanks for the reply Alan.

First reaction - I don't see any of our issues being resolved through the Behave/Gherkin solution. It seems, at best, equivalent to the Enterprise Tester - though possibly more 'custom built tools' required for linkage...??

Simon

Simon,

I simply have to fall back and say that Zephyr doesn't support API connections in its Zephyr for JIRA at the moment. So currently items 1 and 2 are not available as supported functions.

You can see this question has come up before and I've answered in pretty much the same way:

https://answers.atlassian.com/questions/84995/can-i-connect-zephyr-test-case-to-an-automation-script-python-selinium-etc


https://answers.atlassian.com/questions/111199/can-zephyr-for-jira-test-execution-results-be-updated-via-api

Technically speaking, there is another thread or two on Atlassian answers that talks about work-arounds to our lack of company supported APIs. Maybe you can find it using Google.

We are looking at add APIs as an available function to Zephry for JIRA in the future.

Kind Regards,
Chris

Thanks again Chris.

For others'... we've now joined together:

  • Created JIRA Issues (New Feature)
  • Imported JIRA New Features to Zephyr ZCE
  • Created Testcases in ZCE
  • ...and linked to the imported New Features
  • Created defects in ZCE, linked to ZCE Testcase
  • ...and observed the Defect created in JIRA

An alternative flow to the above would be to create JIRA Testcases (or JIRA Zephyr plug-in 'Tests') and import these test definitions (they are only definitions!) to ZCE to link to ZCE Testcases.

Simon

Do we have any update on this. Is it possible for executing test cases using JIRA along with ZBOT ?

Suggest an answer

Log in or Join to answer
Community showcase
Sarah Schuster
Posted Jan 29, 2018 in Jira

What are common themes you've seen across successful & failed Jira Software implementations?

Hey everyone! My name is Sarah Schuster, and I'm a Customer Success Manager in Atlassian specializing in Jira Software Cloud. Over the next few weeks I will be posting discussion topics (8 total) to ...

2,861 views 12 18
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you
Atlassian Team Tour

Join us on the Team Tour

We're bringing product updates and pro tips on teamwork to ten cities around the world.

Save your spot