Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Is it possible to update an existing test execution with a cucumber.json import file

Hello guys,

i have a nightly selenium test running in jenkins. It exports cucumber feature files from Jira/Xray, runs the test and imports the xrayresultimport.json via REST-API back to Jira/Xray.  Currently this creates me a new test execution nightly, where all executed tests are documented as passed or failed.

What i want is, instead of creating a new test execution it should update an already existing test execution night by night and document the tests inside of the test execution as passed or failed.

Here is my xrayresultimport.json

{
"fields": {
"project": {
"key": "MPPITO"
},
"summary": "Test Execution for Cucumber results (Generated by job: <build-tag>) via Pipeline",
"description": "Test Execution for Cucumber results In Jenkins\n(Job Cucumber Report: <build-url>/cucumber-html-reports/overview-features.html) \n Build Job URL: <build-url>",
"customfield_10928": [
"<browser>","Jenkins"
],
"issuetype": {
"id": "10402"
}
}
}

 Here is my rest-api call:

curl -u $ACCESS_KEY_QQCONT1 -F info=@build/xrayresultimport.json -F result=@target/cucumber.json https://.../jira/rest/raven/1.0/import/execution/cucumber/multipart'

 Is this possible or not and how could this work?

Thanks & regards

Max

1 answer

1 accepted

0 votes
Answer accepted

Hi @Maximilian Kaltner ,

for the time being, if you use multipart endpoints, then new Test Execution issues will always be created.

I'm not sure if you're using Jira cloud or server, but nevertheless I've added suggestions to our backlog. Please review and vote on them so we can track your interest on it.

 

https://jira.xpand-it.com/browse/XRAY-6221

https://jira.xpand-it.com/browse/XRAYCLOUD-2375

 

Regards,

Sergio

Hi @Sergio Freire ,

thanks for reply.

Is it possible if i dont use multipart endpoint?

 

Where or how can i vote for https://jira.xpand-it.com/browse/XRAY-6221 and for https://jira.xpand-it.com/browse/XRAYCLOUD-2375 ?

Thanks and best regards
Max

Hi @Maximilian Kaltner ,

 

1. if you dont use the multipart; it's possible you'll need to export the Test Execution firstand the feature file will contain the issue key of the Test Execution. Or you can put it yourself on the feature as a tag on the Feature element.

 

2. you need to login in that Jira instance and then on the right-side pane you should be able to vote, next to the Votes section.

 

Regards,

Sergio

Hi @Sergio Freire

to 1. do I or can I use the same xrayimportresult.json as above or is it another one?

Description of my use-case:

We have a Release-Testplan for each release, there is a linked test-execution with several linked tests . Some of those tests will be done manual and some of it will be done automatically via cucumber/selenium night by night. 

So each night by the automated run, the cucumber testcases of this exeution should get updated if the tests passed or failed.

Regards

Max

Here is my Test-Execution MPPITO-6187 with 3 cucumber Tests:

TESTEXEC.PNGHere is my exported featurefile:

One Zip with 3 featurefiles... all of the featurefiles do include the test execution key:

featurefiles.PNG

 

It runs the cucumber test and afterwards it imports the result with the following commands:

 

log.outputBanner("Import cucumber results")
junit allowEmptyResults: true, keepLongStdio: true, testResults: 'target/**/*.xml'

cucumber 'target/**/*.json'

sh 'sed -i -r s,"<build-tag>","\"${BUILD_TAG}\"",g build/xrayresultimport.json'
sh 'sed -i -r s,"<build-url>","\"${BUILD_URL}\"",g build/xrayresultimport.json'
sh "sed -i -r s,\"<browser>\",\"${params.BROWSER}\",g build/xrayresultimport.json"

sh 'curl -u $ACCESS_KEY_QQCONT1 -F info=@build/xrayresultimport.json -F result=@target/cucumber.json https://myDomain/jira/rest/raven/1.0/import/execution/cucumber'

But the TEST-Execution MPPITO-6187 wont get updated.... aftwards...so what i thought that this testcases getting green or red status is not happening. It is still in "TODO" status.

todo.PNG

Thanks for your help and best regards

Max

Hi @Maximilian Kaltner ,

if you use the standard Cucumber endpoint, as detailed in https://confluence.xpand-it.com/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-CucumberJSONresults, you just have to specify the cucumber json report produced by cucumber.

 

So the curl command would be something like this:

curl -H "Content-Type: application/json" -X POST -u admin:admin --data @target/@cucumber_output.json http://yourserver/rest/raven/1.0/import/execution/cucumber

 

Please try to do this request by hand and check its response to see if everything is fine.

If you did exported the cucumber features from Xray UI or by using the REST API, then the scenarios should be properly tagged with the corresponding Test issues. 

In some instances people configure the export prefix for cucumber Tests with "TEST_".. so a test with issue key CALC-1 would be converted to a tag like @TEST_CALC-1 

Please check also the following setting that you have for replacing hyphens whenever exporting the tags https://confluence.xpand-it.com/display/XRAY/Miscellaneous#Miscellaneous-Replacedash[-]withunderscore[_]whenexportingissuekeysastags

 

Notes: The standard endpoint doesn't allow you to configure the description or other fields on the Test Execution issue though; that's a current limitation that you can only overcome if you do use the multipart endpoint. The multipart endpoint in turn will always create new Test Executions.

 

If you still have doubts, I think the best is to reach out to Xray support and try setting up a call.

Regards,

Sergio

Like rgnair likes this

Hi @Sergio Freire ,

thank you for the hint.

The curl to import is not working in jenkins yet. It could not read data from file, so it sends an empty POST.

Do you see some errors?

2020-07-27 16_47_36-futuremap » ta-test-max #44 Console [Jenkins].png

Thanks a lot for your support

Max

I think you have a typo there with two "@" : @target/@cucumber.json 

It should be just @target/cucumber.json   if the file path is  target/cucumber.json

I think you have a typo there with two "@" : @target/@cucumber.json 

It should be just @target/cucumber.json   if the file path is  target/cucumber.json

Thanks a lot - it was a typo.

It will imported without error now, but my test execution and the linked tests won´t get updated :(

123.png

I think my exported feature files look good, but it is something wrong with the POST maybe...

@Maximilian Kaltner  , please reach out to Xray support and try setting up a call as this needs to be analyzed in further detail and also we may need to analyze your Jira settings.

Regards,

Sergio

Like Maximilian Kaltner likes this

Hi @Sergio Freire ,

thank you so much for you never ending help the last days.

It is working now, so everything is fine - at the end there was a little missing within the curl

curl -H "Content-Type: application/json" -X POST -u admin:admin --data target/@cucumber_output.json http://yourserver/jira/rest/raven/1.0/import/execution/cucumber

Like Sergio Freire likes this

Great to hear that :)

Regards

Sergio

=======================================
Personal blog on testing, Agile and software development: https://sergiofreire.com
Follow me on Twitter: @darktelecom

hi @Sergio Freire 

if you use the standard Cucumber endpoint, as detailed in https://confluence.xpand-it.com/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-CucumberJSONresults, you just have to specify the cucumber json report produced by cucumber.

 

So the curl command would be something like this:

curl -H "Content-Type: application/json" -X POST -u admin:admin --data @target/@cucumber_output.json http://yourserver/rest/raven/1.0/import/execution/cucumber

i just copied your comment on top since i was also experiencing the same issue like @Maximilian Kaltner reported,want to execute multiple test througha single execution key using export cucmber feature...


So if i am using curl -H "Content-Type: application/json" -X POST -u admin:admin --data @target/@cucumber_output.json http://yourserver/rest/raven/1.0/import/execution/cucumber thsi commabd how can i pass my
info=@build/xrayresultimport.json


Hi @rgnair ,

well, that endpoint only accepts the cucumber JSON report.

There is the multipart endpoint (https://docs.getxray.app/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-CucumberJSONresultsMultipart) which accepts the info object. The URL for this endpoint is obviously a distinct one. 

Let me quote the example provided in the docs above.

 

Example Request

curl -u admin:admin -F info=@createTestExec.json -F result=@results.json http://yourserver/rest/raven/1.0/import/execution/cucumber/multipart

 

Hope this clarifies.

If not, let me know :)

Regards

Sergio 

hi @Sergio Freire  thiis what i am using

curl -u admin:admin -F info=@C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json -F result=@C:\CFS-Workspace\testautomation\target\cucumber.json https://########/rest/raven/1.0/import/execution/cucumber/multipart

 

two issues here

1)its not updating the execution key with result creating a new execution key for each test case inside that execution key .it is still in TODO status eventhough its created new execution for these each test case below.Capture.PNGexceution.PNG

@rgnair , as mentioned above, currently if you want to customize the Test Execution issue you'll have to use the multipart endpoint. However, the multipart endpoint will always create new Test Execution issues.

If you don't need to customize the Test Execution issue, you can use the standard cucumber endpoint.

There are currently no workarounds; please vote on the improvement issues I've mentioned above.

In any case, you can always reach out Xray support team and detail exactly your needs.

Regards,

Sérgio

hi @Sergio Freire  thanks for the info.....when i tried this  multi end point url i am getting this error

 

<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>

note: command i used

curl -u XXXXXX -F info=@C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json -F result=@C:\CFS-Workspace\testautomation\target\cucumber.json https://XXXXXXX/rest/raven/1.0/import/execution/cucumber/multipart

@rgnair you should reach out to the Xray support team so they can see this with you in further detail. Probably some permission issues or some authentication issue.

@Sergio Freire its working with normal cucmber json end point with my credentails issue is only with multipoint.So idon't think its an issue with authentication

this is my cucmber json and info file

 

[
{
"line": 2,
"elements": [
{
"start_timestamp": "2020-09-08T13:53:07.348Z",
"before": [
{
"result": {
"duration": 2930184100,
"status": "passed"
},
"match": {
"location": "Hook.InitializeTest(Scenario)"
}
}
],
"line": 6,
"name": "User is able to login",
"description": "",
"id": ";user-is-able-to-login",
"after": [
{
"result": {
"duration": 16476400,
"status": "passed"
},
"match": {
"location": "Hook.TearDownTest(Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 47638800,
"status": "passed"
},
"line": 7,
"name": "I navigate to the Login Page",
"match": {
"location": "WebSteps.java:26"
},
"keyword": "Given "
},
{
"result": {
"error_message": "org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {\"method\":\"css selector\",\"selector\":\"#mainForm\\:userName\"}\n (Session info: chrome\u003d78.0.3904.70)\nFor documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html\nBuild info: version: \u00273.141.59\u0027, revision: \u0027e82be7d358\u0027, time: \u00272018-11-14T08:17:03\u0027\nSystem info: host: \u0027VMUC0901619\u0027, ip: \u002710.80.157.123\u0027, os.name: \u0027Windows 10\u0027, os.arch: \u0027x86\u0027, os.version: \u002710.0\u0027, java.version: \u00271.8.0_151\u0027\nDriver info: org.openqa.selenium.remote.RemoteWebDriver\nCapabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 78.0.3904.70, chrome: {chromedriverVersion: 78.0.3904.70 (edb9c9f3de024..., userDataDir: /tmp/.com.google.Chrome.gpvEUz}, e34: true, e34:executor: 10.8.26.112, e34:version: 1.4.25, e34:video: true, goog:chromeOptions: {debuggerAddress: localhost:40525}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, videoEnabled: true}\nSession ID: fe662ca0-a4f6-43a1-b04b-a66ab82295bb\n*** Element info: {Using\u003did, value\u003dmainForm:userName}\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\r\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:423)\r\n\tat org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)\r\n\tat org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)\r\n\tat org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)\r\n\tat org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:323)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElementById(RemoteWebDriver.java:372)\r\n\tat org.openqa.selenium.By$ById.findElement(By.java:188)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:315)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat org.openqa.selenium.support.events.EventFiringWebDriver.lambda$new$1(EventFiringWebDriver.java:105)\r\n\tat com.sun.proxy.$Proxy14.findElement(Unknown Source)\r\n\tat org.openqa.selenium.support.events.EventFiringWebDriver.findElement(EventFiringWebDriver.java:194)\r\n\tat org.openqa.selenium.support.pagefactory.DefaultElementLocator.findElement(DefaultElementLocator.java:69)\r\n\tat org.openqa.selenium.support.pagefactory.internal.LocatingElementHandler.invoke(LocatingElementHandler.java:38)\r\n\tat com.sun.proxy.$Proxy17.clear(Unknown Source)\r\n\tat acceptancetests.pages.LoginPage.enterUsername(LoginPage.java:45)\r\n\tat acceptancetests.steps.WebSteps.lambda$new$1(WebSteps.java:37)\r\n\tat ✽.I enter \"ADMINPFC\" into field \"Username\"(file:src/test/resources/features/1.feature:8)\r\n",
"duration": 20146800599,
"status": "failed"
},
"line": 8,
"name": "I enter \"ADMINPFC\" into field \"Username\"",
"match": {
"arguments": [
{
"val": "\"ADMINPFC\"",
"offset": 8
},
{
"val": "\"Username\"",
"offset": 30
}
],
"location": "WebSteps.java:34"
},
"keyword": "When "
},
{
"result": {
"duration": 13100,
"status": "skipped"
},
"line": 9,
"name": "I enter \"VKOoTliqY4A9/Bd2tMfjcg\u003d\u003d:kWVdloVbLil8EhVlU9ReSw\u003d\u003d\" into field \"Password\"",
"match": {
"arguments": [
{
"val": "\"VKOoTliqY4A9/Bd2tMfjcg\u003d\u003d:kWVdloVbLil8EhVlU9ReSw\u003d\u003d\"",
"offset": 8
},
{
"val": "\"Password\"",
"offset": 71
}
],
"location": "WebSteps.java:34"
},
"keyword": "When "
},
{
"result": {
"duration": 5300,
"status": "skipped"
},
"line": 10,
"name": "I click the Login Button",
"match": {
"location": "WebSteps.java:50"
},
"keyword": "When "
},
{
"result": {
"duration": 5099,
"status": "skipped"
},
"line": 11,
"name": "the homepage should be visible",
"match": {
"location": "WebSteps.java:59"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@CFROF-19375"
},
{
"name": "@CFROF-11685"
}
]
},
{
"start_timestamp": "2020-09-08T13:53:30.520Z",
"before": [
{
"result": {
"duration": 1719195999,
"status": "passed"
},
"match": {
"location": "Hook.InitializeTest(Scenario)"
}
}
],
"line": 15,
"name": "QuickRejectPayment",
"description": "",
"id": ";quickrejectpayment",
"after": [
{
"result": {
"duration": 13102200,
"status": "passed"
},
"match": {
"location": "Hook.TearDownTest(Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 3308313000,
"status": "passed"
},
"line": 16,
"name": "Login to the application with below \"userName\" and \"password\"",
"match": {
"arguments": [
{
"val": "\"userName\"",
"offset": 36
},
{
"val": "\"password\"",
"offset": 51
}
],
"location": "LoginDefenition.java:21"
},
"rows": [
{
"cells": [
"userName",
"password"
]
},
{
"cells": [
"Qxz09gu",
"pSg8BpX1Gh20LNCTBGEiJA\u003d\u003d:W1lipUKQhdrZWs7yx/q0cBuggFxMtUhrK3HeWGsbQA0\u003d"
]
}
],
"keyword": "Given "
},
{
"result": {
"duration": 3771374800,
"status": "passed"
},
"line": 19,
"name": "Selecting the dealing company as \"FS_DE\"",
"match": {
"arguments": [
{
"val": "\"FS_DE\"",
"offset": 33
}
],
"location": "LoginDefenition.java:31"
},
"keyword": "When "
},
{
"result": {
"duration": 34198099900,
"status": "passed"
},
"line": 20,
"name": "Selecting an eligible transaction",
"match": {
"location": "LoginDefenition.java:37"
},
"keyword": "And "
}
],
"tags": [
{
"name": "@CFROF-19375"
},
{
"name": "@CFROF-19457"
}
]
}
],
"name": "",
"description": "",
"id": "",
"keyword": "Feature",
"uri": "file:src/test/resources/features/1.feature",
"tags": [
{
"name": "@CFROF-19375",
"type": "Tag",
"location": {
"line": 1,
"column": 1
}
}
]
}
]

 

 

Capture.PNGinfo.PNG

 

Thanks in advance for your support :)

@rgnair , I think your info object (xrayresultimport.json) is not correct due to  <build-tag>, <jira-fixed-version-id>, <build-url>.

Try to hardcode values for it first of all.

You should check Jenkins pipeline syntax in order to see how to pass variables; I *think* it should be something like the following, but I'm not 100% sure

 

${myVariable}

hi @Sergio Freire  i have used this endpoint to update the description filed but its giving me wrong results

curl -D- -u xxx:xxx -X PUT --data @C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json https://xxxxxxx/rest/api/2/issue/CFROF-19375

this is what i am getting

HTTP/1.1 405 Not Allowed
Date: Wed, 09 Sep 2020 09:42:34 GMT
Server: nginx/1.11.6
Content-Type: text/html
Content-Length: 173

<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>

 

 

My json data

{
"fields": {
"summary": "data",
"description": "data "
}
}

 

Any idea on how to update a jirs issue field(for me description)

thanks @Sergio Freire  is there anyway i can update the custoem fileds in an issue key(for me description and summary needs to update)

 

i have curled this command and getting same result,any idea about how to get this or how can i update the values 

 

>curl -D- -u xxx:xxx -X PUT --data @C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json https://server/rest/raven/1.0/api/2/issue/CFROF-19375
HTTP/1.1 405 Not Allowed
Date: Wed, 09 Sep 2020 09:42:34 GMT
Server: nginx/1.11.6
Content-Type: text/html
Content-Length: 173

<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>

 

 

my json data

{
"fields": {
"summary": "Test Execution ",
"description": "Test Execution  "
}
}

@rgnair  you should add the Content-Type header in your HTTP PUT request.

 


curl -v -u admin:admin -X PUT -H "Content-Type: application/json" --data @update_issue.json http://192.168.56.102/rest/api/2/issue/CALC-7877

 

 

This is my update_issue.json, as a mere example

 

{
"fields": {
"summary": "Test Execution for cucumber Execution",
"description": "example",
"customfield_11805" : [
"iOS"
],
"fixVersions" :
[
{
"name": "v3.0"
}
]
}

hi @Sergio Freire great and thanks its worked :) but there are diffrenece in the custome field id's it seems.

 

errorMessages":[],"errors":{"fixVersions":"Version name 'v3.0' is not valid"}}* Closing connection 0
* schannel: shutting down SSL/TLS connection with atxxc.xxxxxx.net port 443
* schannel: clear security context handle

When i hit this with decsription only its worked :)...from where can i get the custome fileds ids to pass through this json file,any docs

Yes, please consult ask your Jira admin to provide them, it can be seen in the Jira administration "custom fields" section.

Like rgnair likes this

Besides, the error you're having is because that project doesn't have a release named "v3.0". You need to adjust the request to your scenario.

@Sergio Freire  thank you somuch for the support :)

Like Sergio Freire likes this

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Asked in Jira Service Desk

Calling all Insight users, we need your help!

Hello Insight users,  As part of our (Mindville's) acquisition by Atlassian, our training team is looking to build some new Insight training materials. It would really helpful if you can ...

153 views 1 0
View question

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you