Acceptance Test Driven Development with Jira

Maxime Lemanissier January 9, 2012

Hi there,

We would like to do Acceptance Test Driven Development (ATDD), that is to say, write user stories in a precise format (given / when / then, or equivalent), parseable by a tool to run automated tests (against source code through "fixtures", or against GUI through GUI driver tools like Selenium).

We already tried Fitnesse, Cucumber, etc. but as we're using Jira, we would prefer to have a unique repository / tool for managing our stories.

Is there any Jira plugin that allow to do ATDD, or to integrate with third party ATDD tool?

Thanks,

Maxime

5 answers

2 votes
Bruno Conde
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
July 21, 2014

Hi Maxime,

I suggest Xray for JIRA: https://marketplace.atlassian.com/plugins/com.xpandit.plugins.xray

It's a test management tool supporting both manual and automated tests (using Cucumber) that you can link to requirements (in this case user Stories). You can create automated cucumber tests and generate .feature files to execute externally to JIRA. We are working on a REST API for exporting Cucumber .feature files and importing Test Execution results into JIRA. This will allow you to use CI platforms like Jenkins to execute your tests and have the report and requirement coverage in JIRA.

Best Regards,

Bruno Conde.

edubme February 15, 2017

@Bruno Conde did your team figure out an API to import test execution results into JIRA?

kamalpreet kaur April 19, 2017

@Bruno conde Hi Bruno, We already have tests BDD test running in Cucumber framework, any thing on the API for importing test results back in JIRA? Went through a lot of documentation online but no luck 

Nick Iles _AssertThat_ February 18, 2019

Hi, 

We've built a couple of frameworks over the last few years and found that all the solutions had a few gaps in functionality so we decided to build our own.  We released it a couple of weeks ago and are building up a good client base.   The plugin includes a cucumber.json import so your test results are automatically imported back into Jira.  This allows you to see your test results alongside your user storied.

We also built in some functionality to analyse your test results so you can assign defects etc all in Jira.

Please take a look and we'd really appreciate your feedback.

https://marketplace.atlassian.com/apps/1219033/assertthat-bdd-test-management-in-jira?hosting=cloud&tab=overview

Thanks

Nick

Like Glib likes this
0 votes
Nick November 3, 2013

Maxime - We're going to try this as well and we've looked at Behave but aren't sure we like it. The biggest limitation was being able to group and categorise stories/acceptance critiera.

Instead we're going to try using Bob Swifts CLI tool to export BDD/ATDD scripts from the description fields of JIRA User Stories to text files. We will then try and pump them through Cucumber bolted onto a Selenium driver and into our application.

Wish us luck!

Emidio Stani July 1, 2014

@Nick

how did it go with the bob swift cli tool ? I might think to implement the same thing, for example launching jenkins, which retrieves the user stories from jira and then execute cucumber to launch selenium driver.

Nikhil April 12, 2018

Hi @Maxime Lemaniss1@Bruno Conde @kamalpreet kaur@edubme@Emidio Stani

Actually I am totally new for XRAY- Cucumber integration. Basically I have one cucumber bdd .feature file written in Intellij IDE. At the same time I have corresponding step definition too which is working fine locally. Now as per my requirement, I need to trigger this feature from Xray-Jira but don't know how to do that, I mean which api to call, how to call that api, where to call that api etc etc.

I tried to find some good document on google but could not find anything fruitful which can help me step by step setup.

Can anyone please help me in this ?

Thanks.

Regards,

Nikhil Agrawal

Bruno Conde
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
April 13, 2018

Hi Nikhil,

 

You can define cucumber tests directly in Xray.  For this you can just create a new Test issue and choose the Cucumber type. 

A cucumber test in xray only contains the scenario definition. You should create a Test issue per scenario.

The implementation part, or the code that will parse the scenario and execute it's steps is not meant to be stored in Jira. You can use your favorite code repository. 

Xray also provides an API that allow you to import your cucumber feature files. This will automatically create Tests, and other entities in Jira:

https://confluence.xpand-addons.com/display/XRAY/Importing+Cucumber+Tests+-+REST

This is just for defining/importing ccuumber tests in Xray. Regarding the import of execution results Xray also provides a REST API that allows you to simply take your cucumber results json file and import it directly to Jira. This will update/create new Test Executions in Xray with the results. Remember that it is not the purpose of Xray/Jira to execute cucumber tests. Xray just stores the execution results and outputs nice reports.

It is possible to configure builds in a CI tool that will:

1. Export the cucumber definitions from Xray issues:

2. Checkout your implementation from the repository and merge it with the feature files exported in step 1.

3. Execute the tests with cucumber

4. Import the execution results into Xray.

 

Xray provides plugins for Jenkins and Bamboo to easy the configuration of these tasks.

It is also possible to import the cucumber execution results right from the UI (from a Test Execution or a Test Run).

Please check the following links for importing execution results into Xray:

https://confluence.xpand-addons.com/display/XRAY/Import+Execution+Results+-+REST

 https://confluence.xpand-addons.com/display/XRAY/Continuous+Integration+Tools

https://confluence.xpand-addons.com/display/XRAY/Integration+with+Bamboo

https://confluence.xpand-addons.com/display/XRAY/Integration+with+Jenkins

https://confluence.xpand-addons.com/display/XRAY/Exporting+Cucumber+Tests+-+REST

 

Hope this helps,

Regards,

Bruno Conde

Nikhil April 13, 2018

Thanks a lot @Bruno Conde. This information really helped me a lot. Just want to confirm one thing, do we have any plugin of Jira-XRAY with Bitbucket code repository or it would be handled other way round, I mean plugins needs to be installed in Jenkins (CI tool).

What I understood is, we do not have any bitbucket or jenkins plugins that can be configured in XRay rather we can install these XRay and Bitbucket plugins in Jenkins. Correct me if I am wrong ?

Bruno Conde
Marketplace Partner
Marketplace Partners provide apps and integrations available on the Atlassian Marketplace that extend the power of Atlassian products.
November 14, 2018

Hi @Nikhil,

Sorry for the late response but somehow I have missed the notification for this answer.

The import execution results process is driven by the CI platform. Xray provides plugins for the most common CI platforms like Jenkins, Bamboo, TeamCity.

These plugins will just help you call the Xray rest API to import execution results.

Hence, you configure the integration between Xray and CI on the CI side.

Regards,

Bruno Conde

0 votes
ILYUNG July 15, 2013

I have tried "Behave for JIRA" but there is some limitation in the features. For example, users can not search the created features. And it seems to be hard to organize the created features as well. And what are the 'tags' for?

@Alan Parkinson, do you have some guide or solution for those?

0 votes
Jeremy Largman
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 12, 2012

Our documentation is a bit outdated, but there's Using JIRA for Test Case Management. Check that out and its sub-pages. In general, search Google (and our resources, like confluence.atlassian.com) for 'test case management'. There's also a good thread over on Stack Overflow.

Maxime Lemanissier January 13, 2012

ATDD is about test automation. You define user stories as test scenario, and you make them runnable on your system. It's like unit tests understandable and writable by business analysts.

Greenpepper, which is based on Confluence is a tool of that kind. As confluence and jira are integrated, this could be a solution, but a complicated and expensive one (2 new tools and licenses), considering a jira plugin could do the work.

Alan Parkinson [Hindsight Software]
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
July 22, 2012

We have built such a JIRA Plugin (Behave for JIRA) and its now available in the marketplace https://marketplace.atlassian.com/plugins/com.hindsighttesting.behave.jira

It allows "Scenarios" to be added to user stories or any JIRA issue as Acceptance Tests. The scenarios are written in Cucumber's Gherkin format and a maven plugin transforms the relevant scenarios into features files during the build. Cucumber can then run the generate feature files to execute the tests.

0 votes
Renjith Pillai
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 9, 2012

I am not sure whether you need a plugin for this as this is more of a practice that needs to be followed in the project. This is what we used to follow. We used to have an issuetype called testcase and with some custom fields like 'pre-condition', 'post-condition' to fully describe the testcase. So for every userstory there will be a set of testcases which are linked to the userstory using the Jira linking function. The link that was used was 'is tested by' even though the type of link has no relevance apart from correcting displaying the relation while viewing the story.

The testcase issue type also was associated with a workflow which moves from states New => Under Testing => Success/Failure

Additionally, our CI system automatically transistions the testcase to success when the testcase passes in the CI and this becomes the closure DONE criteria for the userstory.

Hope this helps.

Maxime Lemanissier January 13, 2012

how do you link the testcase in jira and the testcase run in ci server ? using jira api?

Renjith Pillai
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 13, 2012

Yes, access jira using SOAP interface.

Suggest an answer

Log in or Sign up to answer