Acceptance Test Driven Development with Jira

Hi there,

We would like to do Acceptance Test Driven Development (ATDD), that is to say, write user stories in a precise format (given / when / then, or equivalent), parseable by a tool to run automated tests (against source code through "fixtures", or against GUI through GUI driver tools like Selenium).

We already tried Fitnesse, Cucumber, etc. but as we're using Jira, we would prefer to have a unique repository / tool for managing our stories.

Is there any Jira plugin that allow to do ATDD, or to integrate with third party ATDD tool?

Thanks,

Maxime

5 answers

Hi Maxime,

I suggest Xray for JIRA: https://marketplace.atlassian.com/plugins/com.xpandit.plugins.xray

It's a test management tool supporting both manual and automated tests (using Cucumber) that you can link to requirements (in this case user Stories). You can create automated cucumber tests and generate .feature files to execute externally to JIRA. We are working on a REST API for exporting Cucumber .feature files and importing Test Execution results into JIRA. This will allow you to use CI platforms like Jenkins to execute your tests and have the report and requirement coverage in JIRA.

Best Regards,

Bruno Conde.

@Bruno Conde did your team figure out an API to import test execution results into JIRA?

@Bruno conde Hi Bruno, We already have tests BDD test running in Cucumber framework, any thing on the API for importing test results back in JIRA? Went through a lot of documentation online but no luck 

I am not sure whether you need a plugin for this as this is more of a practice that needs to be followed in the project. This is what we used to follow. We used to have an issuetype called testcase and with some custom fields like 'pre-condition', 'post-condition' to fully describe the testcase. So for every userstory there will be a set of testcases which are linked to the userstory using the Jira linking function. The link that was used was 'is tested by' even though the type of link has no relevance apart from correcting displaying the relation while viewing the story.

The testcase issue type also was associated with a workflow which moves from states New => Under Testing => Success/Failure

Additionally, our CI system automatically transistions the testcase to success when the testcase passes in the CI and this becomes the closure DONE criteria for the userstory.

Hope this helps.

how do you link the testcase in jira and the testcase run in ci server ? using jira api?

Yes, access jira using SOAP interface.

0 votes

Our documentation is a bit outdated, but there's Using JIRA for Test Case Management. Check that out and its sub-pages. In general, search Google (and our resources, like confluence.atlassian.com) for 'test case management'. There's also a good thread over on Stack Overflow.

ATDD is about test automation. You define user stories as test scenario, and you make them runnable on your system. It's like unit tests understandable and writable by business analysts.

Greenpepper, which is based on Confluence is a tool of that kind. As confluence and jira are integrated, this could be a solution, but a complicated and expensive one (2 new tools and licenses), considering a jira plugin could do the work.

We have built such a JIRA Plugin (Behave for JIRA) and its now available in the marketplace https://marketplace.atlassian.com/plugins/com.hindsighttesting.behave.jira

It allows "Scenarios" to be added to user stories or any JIRA issue as Acceptance Tests. The scenarios are written in Cucumber's Gherkin format and a maven plugin transforms the relevant scenarios into features files during the build. Cucumber can then run the generate feature files to execute the tests.

I have tried "Behave for JIRA" but there is some limitation in the features. For example, users can not search the created features. And it seems to be hard to organize the created features as well. And what are the 'tags' for?

@Alan Parkinson, do you have some guide or solution for those?

Maxime - We're going to try this as well and we've looked at Behave but aren't sure we like it. The biggest limitation was being able to group and categorise stories/acceptance critiera.

Instead we're going to try using Bob Swifts CLI tool to export BDD/ATDD scripts from the description fields of JIRA User Stories to text files. We will then try and pump them through Cucumber bolted onto a Selenium driver and into our application.

Wish us luck!

@Nick

how did it go with the bob swift cli tool ? I might think to implement the same thing, for example launching jenkins, which retrieves the user stories from jira and then execute cucumber to launch selenium driver.

Hi @Maxime Lemaniss1@Bruno Conde [Xpand IT] @kamalpreet kaur@Evan Walker Wil@Emidio Stani

Actually I am totally new for XRAY- Cucumber integration. Basically I have one cucumber bdd .feature file written in Intellij IDE. At the same time I have corresponding step definition too which is working fine locally. Now as per my requirement, I need to trigger this feature from Xray-Jira but don't know how to do that, I mean which api to call, how to call that api, where to call that api etc etc.

I tried to find some good document on google but could not find anything fruitful which can help me step by step setup.

Can anyone please help me in this ?

Thanks.

Regards,

Nikhil Agrawal

Hi Nikhil,

 

You can define cucumber tests directly in Xray.  For this you can just create a new Test issue and choose the Cucumber type. 

A cucumber test in xray only contains the scenario definition. You should create a Test issue per scenario.

The implementation part, or the code that will parse the scenario and execute it's steps is not meant to be stored in Jira. You can use your favorite code repository. 

Xray also provides an API that allow you to import your cucumber feature files. This will automatically create Tests, and other entities in Jira:

https://confluence.xpand-addons.com/display/XRAY/Importing+Cucumber+Tests+-+REST

This is just for defining/importing ccuumber tests in Xray. Regarding the import of execution results Xray also provides a REST API that allows you to simply take your cucumber results json file and import it directly to Jira. This will update/create new Test Executions in Xray with the results. Remember that it is not the purpose of Xray/Jira to execute cucumber tests. Xray just stores the execution results and outputs nice reports.

It is possible to configure builds in a CI tool that will:

1. Export the cucumber definitions from Xray issues:

2. Checkout your implementation from the repository and merge it with the feature files exported in step 1.

3. Execute the tests with cucumber

4. Import the execution results into Xray.

 

Xray provides plugins for Jenkins and Bamboo to easy the configuration of these tasks.

It is also possible to import the cucumber execution results right from the UI (from a Test Execution or a Test Run).

Please check the following links for importing execution results into Xray:

https://confluence.xpand-addons.com/display/XRAY/Import+Execution+Results+-+REST

 https://confluence.xpand-addons.com/display/XRAY/Continuous+Integration+Tools

https://confluence.xpand-addons.com/display/XRAY/Integration+with+Bamboo

https://confluence.xpand-addons.com/display/XRAY/Integration+with+Jenkins

https://confluence.xpand-addons.com/display/XRAY/Exporting+Cucumber+Tests+-+REST

 

Hope this helps,

Regards,

Bruno Conde

Thanks a lot @Bruno Conde [Xpand IT]. This information really helped me a lot. Just want to confirm one thing, do we have any plugin of Jira-XRAY with Bitbucket code repository or it would be handled other way round, I mean plugins needs to be installed in Jenkins (CI tool).

What I understood is, we do not have any bitbucket or jenkins plugins that can be configured in XRay rather we can install these XRay and Bitbucket plugins in Jenkins. Correct me if I am wrong ?

Hi @Nikhil,

Sorry for the late response but somehow I have missed the notification for this answer.

The import execution results process is driven by the CI platform. Xray provides plugins for the most common CI platforms like Jenkins, Bamboo, TeamCity.

These plugins will just help you call the Xray rest API to import execution results.

Hence, you configure the integration between Xray and CI on the CI side.

Regards,

Bruno Conde

Suggest an answer

Log in or Sign up to answer
Community showcase
Posted Monday in Jira

Meet the Jira Software 8.0 platform release

  Hello Atlassian community!   We're excited to share the Jira Server and Data Center 8.0 platform release.  With 8.0, teams will be able to work more productively and sca...

787 views 15 12
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you