How do I execute automated tests from JIRA? [Zephyr]

[Simon Hammond]

Sorry to be slow...

We've just been playing with Zephyr with the idea that an automated stack could be set up as follows:

• Zephyr plug-in for JIRA (write test cases linked to requirements)
• Zephyr community or enterprise edition (automatically sync with Zephyr plug-in Tests, then further details added such as link to ZBOTs and scripts)
• ZBOT deployed on to system under test
• C# running script on system under test

We were hoping the following:

• The 'EXECUTE' button in the Zephyr plug-in would actually 'execute' a Test through the above stack and render the result directly back to the Zephyr plug-in within JIRA.
• This would allow:

1. BAs to create test cases as part of their New Feature/Stories elaboration
2. Direct linkage of New Features/Stories to Test issues (the 'Test' being the Zephyr plug-in Issue type)
3. Direct linkage of Test results through to JIRA Defects

I feel like I lost the plot somewhere as the 'EXECUTE' button in Zephyr plug-in only seems to be a manual field. Though the above objective seems intuitively reasonable!

Questions...

a) Can Zephyr achieve what we'd like to do? (i.e. direct execution of automated tests from JIRA and result passed back to JIRA)

b) If not, why not... it seems a reasonable objective?

c) Is there another product which would allow this?

Many thanks for any advice.

[Chris Miller · Accepted solution]

Hi Simon,

Zephyr here. It sounds like you're merging the performance of two of our products in your mind. Let me explain.

Questions...

a) Can Zephyr achieve what we'd like to do? (i.e. direct execution of automated tests from JIRA and result passed back to JIRA)

Not yet.

Currently the Zephyr Enterprise Edition/Community Edition are unique products that don't interact with and are sold seperatly from the Zephyr for JIRA plugin. (Note they do come with a Zephyr Enterprise Connector Plugin, which offers different functionality than the ZFJ plugin.)

These two products do have the Zbot you are speaking of. When tests are executed from ZEE or ZCE, the results come back into ZEE or ZCE. Both of these products offer great integration with JIRA. For example, on the Zephyr Requirement side, you can see current content of any user story or issue you "import" (where we import only pointers to the current content) from JIRA. As well, you can create, update and edit defects or other issues from the Zephyr Defect Tracking UI which are communicated directly to the JIRA database. So there is no syncronization going on at all. As well, ZEE and ZCE integrate with both JIRA On-Demand and JIRA On-Premise Deployments.

As well, ZEE and ZCE come with APIs which can be perfect for CI scenarios. Many of our customers do use the ZEE APIs for CI.

The Zephyr for JIRA plugin doesn't yet offer APIs as a publically supported feature.

b) If not, why not... it seems a reasonable objective?

Yes it does. APIs for Zephyr for JIRA are on our roadmap and we'll be excited to offer them. This is already a highly requested feature.

Follow our Blog for the any and all current announcements about our product: http://blogs.yourzephyr.com/

Kind Regards,

Chris

[Simon Hammond]

Thanks Chris.

Yep - we got there today & realised that the 2 products (Zephyr plug-in & ZEE/ZCE) didn't join up. Our specific challenges are:

• We want our BAs to start to generate requirements and associated tests. For this reason it would add to complexity (and act as a barrier) if we presented them with 2 different looks for requirements & tests => there is a strong incentive to keep both in JIRA and therefore use Zephyr plug-in rather than ZEE/ZCE.
• We now understand it is possible to use the Zephyr plug-in along with the JIRA REST APIs & Bamboo to construct a more hybrid CI/auto-test model but I guess it has the following limitations..?:

1. 'Execute' in the Zephyr plug-in will still not trigger anything to run? i.e. there is no way of generating an external command from the Zephyr plug-in?
2. We can trigger execution from outside JIRA and then use the JIRA API to update the PASS/FAIL on individual tests? [Note we would want to use the test cycle functionality to differentiate between regression test runs etc.] This would avoid manually changing the status of all tests after a run.
3. It would need some [much?] work to construct this stack..?

Any further comments much appreciated.

Thanks for your advice.

Simon

[Daniel]

Hi Everyone,

Just to update this thread one last time, ZAPI is now available. ZAPI adds API capabilities to Zephyr for JIRA. You can find the ZAPI Marketplace page by following the link below.

https://marketplace.atlassian.com/plugins/com.thed.zephyr.zapi

Regards,

Daniel

[Raja kumar]

Do we have any update on this. Is it possible for executing test cases using JIRA along with ZBOT ?

[Bart De Baere]

Two years later... is Simon's use case possible today (using Zephyr)?

1. BAs to create test cases as part of their New Feature/Stories elaboration
2. Direct linkage of New Features/Stories to Test issues (the 'Test' being the Zephyr plug-in Issue type)
3. Direct linkage of Test results through to JIRA (eg. failed/passed)

[Daniel]

Hi Bart, Yes, we now have the ZAPI plugin. This plugin is purchased alongside Zephyr for JIRA Server and gives it a full suite of powerful REST API. You can use JIRA's API to create a Test issue and then ZAPI to create the steps in that issue. You can issue link Tests to another other issue type You can also use the ZAPI REST API to bring in automation results directly into Zephyr for JIRA For more information on ZAPI, please see full details here: https://marketplace.atlassian.com/plugins/com.thed.zephyr.zapi/server/overview Regards, Daniel

[Simon Hammond]

Thanks again Chris.

For others'... we've now joined together:

• Created JIRA Issues (New Feature)
• Imported JIRA New Features to Zephyr ZCE
• Created Testcases in ZCE
• ...and linked to the imported New Features
• Created defects in ZCE, linked to ZCE Testcase
• ...and observed the Defect created in JIRA

An alternative flow to the above would be to create JIRA Testcases (or JIRA Zephyr plug-in 'Tests') and import these test definitions (they are only definitions!) to ZCE to link to ZCE Testcases.

Simon

[Chris Miller]

Simon,

I simply have to fall back and say that Zephyr doesn't support API connections in its Zephyr for JIRA at the moment. So currently items 1 and 2 are not available as supported functions.

You can see this question has come up before and I've answered in pretty much the same way:

https://answers.atlassian.com/questions/84995/can-i-connect-zephyr-test-case-to-an-automation-script-python-selinium-etc

https://answers.atlassian.com/questions/111199/can-zephyr-for-jira-test-execution-results-be-updated-via-api

Technically speaking, there is another thread or two on Atlassian answers that talks about work-arounds to our lack of company supported APIs. Maybe you can find it using Google.

We are looking at add APIs as an available function to Zephry for JIRA in the future.

Kind Regards,
Chris

[Alan Parkinson [Hindsight Software]]

Hi Simon

Have you consider using Behave for JIRA?

All Accetance Tests related to JIRA issues are downloaded from JIRA using custom build tools and then you can use any Gherkin based Tests tool for execution. For C# this would be SpecFlow would be the Gherkin based tool of choice.

Test results wouldn't be displayed in JIRA but through your CI system of choice.

Alan

[Simon Hammond]

Thanks for the reply Alan.

First reaction - I don't see any of our issues being resolved through the Behave/Gherkin solution. It seems, at best, equivalent to the Enterprise Tester - though possibly more 'custom built tools' required for linkage...??

Simon

[Simon Hammond]

I've now watched a few videos of Enterprise Tester to see if it gives us the missing automation directly from JIRA. It doesn't seem to...

• It can bi-directionally synchronise on selected Issue types (and Issue fields) from JIRA: therefore a bespoke Test* Issue type in JIRA could be used as an import type to E.T.
• Execution would be in Enterprise Tester
• The bi-directional update suggests results could be populated back into JIRA Tests* - and our workflow managed through JIRA Issue types Stories/Tests*/Defects
• (*not a native type)

So the advertised 'integration' with JIRA seems more about data exchange than runtime execution - for both tools. As both tools also have requirement and defect types I'm beginning to wonder why bother with JIRA at all... i.e. why not use one of these to manage the entire workflow?