Hello guys,
i have a nightly selenium test running in jenkins. It exports cucumber feature files from Jira/Xray, runs the test and imports the xrayresultimport.json via REST-API back to Jira/Xray. Currently this creates me a new test execution nightly, where all executed tests are documented as passed or failed.
What i want is, instead of creating a new test execution it should update an already existing test execution night by night and document the tests inside of the test execution as passed or failed.
Here is my xrayresultimport.json
{
"fields": {
"project": {
"key": "MPPITO"
},
"summary": "Test Execution for Cucumber results (Generated by job: <build-tag>) via Pipeline",
"description": "Test Execution for Cucumber results In Jenkins\n(Job Cucumber Report: <build-url>/cucumber-html-reports/overview-features.html) \n Build Job URL: <build-url>",
"customfield_10928": [
"<browser>","Jenkins"
],
"issuetype": {
"id": "10402"
}
}
}
Here is my rest-api call:
curl -u $ACCESS_KEY_QQCONT1 -F info=@build/xrayresultimport.json -F result=@target/cucumber.json https://.../jira/rest/raven/1.0/import/execution/cucumber/multipart'
Is this possible or not and how could this work?
Thanks & regards
Max
Hi @Maximilian Kaltner ,
for the time being, if you use multipart endpoints, then new Test Execution issues will always be created.
I'm not sure if you're using Jira cloud or server, but nevertheless I've added suggestions to our backlog. Please review and vote on them so we can track your interest on it.
https://jira.xpand-it.com/browse/XRAY-6221
https://jira.xpand-it.com/browse/XRAYCLOUD-2375
Regards,
Sergio
thanks for reply.
Is it possible if i dont use multipart endpoint?
Where or how can i vote for https://jira.xpand-it.com/browse/XRAY-6221 and for https://jira.xpand-it.com/browse/XRAYCLOUD-2375 ?
Thanks and best regards
Max
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @Maximilian Kaltner ,
1. if you dont use the multipart; it's possible you'll need to export the Test Execution firstand the feature file will contain the issue key of the Test Execution. Or you can put it yourself on the feature as a tag on the Feature element.
2. you need to login in that Jira instance and then on the right-side pane you should be able to vote, next to the Votes section.
Regards,
Sergio
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
to 1. do I or can I use the same xrayimportresult.json as above or is it another one?
Description of my use-case:
We have a Release-Testplan for each release, there is a linked test-execution with several linked tests . Some of those tests will be done manual and some of it will be done automatically via cucumber/selenium night by night.
So each night by the automated run, the cucumber testcases of this exeution should get updated if the tests passed or failed.
Regards
Max
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Here is my Test-Execution MPPITO-6187 with 3 cucumber Tests:
Here is my exported featurefile:
One Zip with 3 featurefiles... all of the featurefiles do include the test execution key:
It runs the cucumber test and afterwards it imports the result with the following commands:
log.outputBanner("Import cucumber results")
junit allowEmptyResults: true, keepLongStdio: true, testResults: 'target/**/*.xml'
cucumber 'target/**/*.json'
sh 'sed -i -r s,"<build-tag>","\"${BUILD_TAG}\"",g build/xrayresultimport.json'
sh 'sed -i -r s,"<build-url>","\"${BUILD_URL}\"",g build/xrayresultimport.json'
sh "sed -i -r s,\"<browser>\",\"${params.BROWSER}\",g build/xrayresultimport.json"
sh 'curl -u $ACCESS_KEY_QQCONT1 -F info=@build/xrayresultimport.json -F result=@target/cucumber.json https://myDomain/jira/rest/raven/1.0/import/execution/cucumber'
But the TEST-Execution MPPITO-6187 wont get updated.... aftwards...so what i thought that this testcases getting green or red status is not happening. It is still in "TODO" status.
Thanks for your help and best regards
Max
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @Maximilian Kaltner ,
if you use the standard Cucumber endpoint, as detailed in https://confluence.xpand-it.com/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-CucumberJSONresults, you just have to specify the cucumber json report produced by cucumber.
So the curl command would be something like this:
curl -H "Content-Type: application/json" -X POST -u admin:admin --data @target/@cucumber_output.json http://yourserver/rest/raven/1.0/import/execution/cucumber
Please try to do this request by hand and check its response to see if everything is fine.
If you did exported the cucumber features from Xray UI or by using the REST API, then the scenarios should be properly tagged with the corresponding Test issues.
In some instances people configure the export prefix for cucumber Tests with "TEST_".. so a test with issue key CALC-1 would be converted to a tag like @TEST_CALC-1
Please check also the following setting that you have for replacing hyphens whenever exporting the tags https://confluence.xpand-it.com/display/XRAY/Miscellaneous#Miscellaneous-Replacedash[-]withunderscore[_]whenexportingissuekeysastags
Notes: The standard endpoint doesn't allow you to configure the description or other fields on the Test Execution issue though; that's a current limitation that you can only overcome if you do use the multipart endpoint. The multipart endpoint in turn will always create new Test Executions.
If you still have doubts, I think the best is to reach out to Xray support and try setting up a call.
Regards,
Sergio
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
thank you for the hint.
The curl to import is not working in jenkins yet. It could not read data from file, so it sends an empty POST.
Do you see some errors?
Thanks a lot for your support
Max
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I think you have a typo there with two "@" : @target/@cucumber.json
It should be just @target/cucumber.json if the file path is target/cucumber.json
I think you have a typo there with two "@" : @target/@cucumber.json
It should be just @target/cucumber.json if the file path is target/cucumber.json
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thanks a lot - it was a typo.
It will imported without error now, but my test execution and the linked tests won´t get updated :(
I think my exported feature files look good, but it is something wrong with the POST maybe...
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Maximilian Kaltner , please reach out to Xray support and try setting up a call as this needs to be analyzed in further detail and also we may need to analyze your Jira settings.
Regards,
Sergio
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
thank you so much for you never ending help the last days.
It is working now, so everything is fine - at the end there was a little missing within the curl
curl -H "Content-Type: application/json" -X POST -u admin:admin --data target/@cucumber_output.json http://yourserver/jira/rest/raven/1.0/import/execution/cucumber
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Great to hear that :)
Regards
Sergio
=======================================
Personal blog on testing, Agile and software development: https://sergiofreire.com
Follow me on Twitter: @darktelecom
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
if you use the standard Cucumber endpoint, as detailed in https://confluence.xpand-it.com/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-CucumberJSONresults, you just have to specify the cucumber json report produced by cucumber.
So the curl command would be something like this:
curl -H "Content-Type: application/json" -X POST -u admin:admin --data @target/@cucumber_output.json http://yourserver/rest/raven/1.0/import/execution/cucumber
i just copied your comment on top since i was also experiencing the same issue like @Maximilian Kaltner reported,want to execute multiple test througha single execution key using export cucmber feature...
So if i am using curl -H "Content-Type: application/json" -X POST -u admin:admin --data @target/@cucumber_output.json http://yourserver/rest/raven/1.0/import/execution/cucumber thsi commabd how can i pass my
info=@build/xrayresultimport.json
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @rgnair ,
well, that endpoint only accepts the cucumber JSON report.
There is the multipart endpoint (https://docs.getxray.app/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-CucumberJSONresultsMultipart) which accepts the info object. The URL for this endpoint is obviously a distinct one.
Let me quote the example provided in the docs above.
Example Request
curl -u admin:admin -F info=@createTestExec.json -F result=@results.json http://yourserver/rest/raven/1.0/import/execution/cucumber/multipart
Hope this clarifies.
If not, let me know :)
Regards
Sergio
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
hi @Sergio Freire - Xblend thiis what i am using
curl -u admin:admin -F info=@C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json -F result=@C:\CFS-Workspace\testautomation\target\cucumber.json https://########/rest/raven/1.0/import/execution/cucumber/multipart
two issues here
1)its not updating the execution key with result creating a new execution key for each test case inside that execution key .it is still in TODO status eventhough its created new execution for these each test case below.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@rgnair , as mentioned above, currently if you want to customize the Test Execution issue you'll have to use the multipart endpoint. However, the multipart endpoint will always create new Test Execution issues.
If you don't need to customize the Test Execution issue, you can use the standard cucumber endpoint.
There are currently no workarounds; please vote on the improvement issues I've mentioned above.
In any case, you can always reach out Xray support team and detail exactly your needs.
Regards,
Sérgio
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
hi @Sergio Freire - Xblend thanks for the info.....when i tried this multi end point url i am getting this error
<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>
note: command i used
curl -u XXXXXX -F info=@C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json -F result=@C:\CFS-Workspace\testautomation\target\cucumber.json https://XXXXXXX/rest/raven/1.0/import/execution/cucumber/multipart
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@rgnair you should reach out to the Xray support team so they can see this with you in further detail. Probably some permission issues or some authentication issue.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@Sergio Freire - Xblend its working with normal cucmber json end point with my credentails issue is only with multipoint.So idon't think its an issue with authentication
this is my cucmber json and info file
[
{
"line": 2,
"elements": [
{
"start_timestamp": "2020-09-08T13:53:07.348Z",
"before": [
{
"result": {
"duration": 2930184100,
"status": "passed"
},
"match": {
"location": "Hook.InitializeTest(Scenario)"
}
}
],
"line": 6,
"name": "User is able to login",
"description": "",
"id": ";user-is-able-to-login",
"after": [
{
"result": {
"duration": 16476400,
"status": "passed"
},
"match": {
"location": "Hook.TearDownTest(Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 47638800,
"status": "passed"
},
"line": 7,
"name": "I navigate to the Login Page",
"match": {
"location": "WebSteps.java:26"
},
"keyword": "Given "
},
{
"result": {
"error_message": "org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {\"method\":\"css selector\",\"selector\":\"#mainForm\\:userName\"}\n (Session info: chrome\u003d78.0.3904.70)\nFor documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html\nBuild info: version: \u00273.141.59\u0027, revision: \u0027e82be7d358\u0027, time: \u00272018-11-14T08:17:03\u0027\nSystem info: host: \u0027VMUC0901619\u0027, ip: \u002710.80.157.123\u0027, os.name: \u0027Windows 10\u0027, os.arch: \u0027x86\u0027, os.version: \u002710.0\u0027, java.version: \u00271.8.0_151\u0027\nDriver info: org.openqa.selenium.remote.RemoteWebDriver\nCapabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 78.0.3904.70, chrome: {chromedriverVersion: 78.0.3904.70 (edb9c9f3de024..., userDataDir: /tmp/.com.google.Chrome.gpvEUz}, e34: true, e34:executor: 10.8.26.112, e34:version: 1.4.25, e34:video: true, goog:chromeOptions: {debuggerAddress: localhost:40525}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, videoEnabled: true}\nSession ID: fe662ca0-a4f6-43a1-b04b-a66ab82295bb\n*** Element info: {Using\u003did, value\u003dmainForm:userName}\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\r\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:423)\r\n\tat org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)\r\n\tat org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)\r\n\tat org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)\r\n\tat org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:323)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElementById(RemoteWebDriver.java:372)\r\n\tat org.openqa.selenium.By$ById.findElement(By.java:188)\r\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:315)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat org.openqa.selenium.support.events.EventFiringWebDriver.lambda$new$1(EventFiringWebDriver.java:105)\r\n\tat com.sun.proxy.$Proxy14.findElement(Unknown Source)\r\n\tat org.openqa.selenium.support.events.EventFiringWebDriver.findElement(EventFiringWebDriver.java:194)\r\n\tat org.openqa.selenium.support.pagefactory.DefaultElementLocator.findElement(DefaultElementLocator.java:69)\r\n\tat org.openqa.selenium.support.pagefactory.internal.LocatingElementHandler.invoke(LocatingElementHandler.java:38)\r\n\tat com.sun.proxy.$Proxy17.clear(Unknown Source)\r\n\tat acceptancetests.pages.LoginPage.enterUsername(LoginPage.java:45)\r\n\tat acceptancetests.steps.WebSteps.lambda$new$1(WebSteps.java:37)\r\n\tat ✽.I enter \"ADMINPFC\" into field \"Username\"(file:src/test/resources/features/1.feature:8)\r\n",
"duration": 20146800599,
"status": "failed"
},
"line": 8,
"name": "I enter \"ADMINPFC\" into field \"Username\"",
"match": {
"arguments": [
{
"val": "\"ADMINPFC\"",
"offset": 8
},
{
"val": "\"Username\"",
"offset": 30
}
],
"location": "WebSteps.java:34"
},
"keyword": "When "
},
{
"result": {
"duration": 13100,
"status": "skipped"
},
"line": 9,
"name": "I enter \"VKOoTliqY4A9/Bd2tMfjcg\u003d\u003d:kWVdloVbLil8EhVlU9ReSw\u003d\u003d\" into field \"Password\"",
"match": {
"arguments": [
{
"val": "\"VKOoTliqY4A9/Bd2tMfjcg\u003d\u003d:kWVdloVbLil8EhVlU9ReSw\u003d\u003d\"",
"offset": 8
},
{
"val": "\"Password\"",
"offset": 71
}
],
"location": "WebSteps.java:34"
},
"keyword": "When "
},
{
"result": {
"duration": 5300,
"status": "skipped"
},
"line": 10,
"name": "I click the Login Button",
"match": {
"location": "WebSteps.java:50"
},
"keyword": "When "
},
{
"result": {
"duration": 5099,
"status": "skipped"
},
"line": 11,
"name": "the homepage should be visible",
"match": {
"location": "WebSteps.java:59"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@CFROF-19375"
},
{
"name": "@CFROF-11685"
}
]
},
{
"start_timestamp": "2020-09-08T13:53:30.520Z",
"before": [
{
"result": {
"duration": 1719195999,
"status": "passed"
},
"match": {
"location": "Hook.InitializeTest(Scenario)"
}
}
],
"line": 15,
"name": "QuickRejectPayment",
"description": "",
"id": ";quickrejectpayment",
"after": [
{
"result": {
"duration": 13102200,
"status": "passed"
},
"match": {
"location": "Hook.TearDownTest(Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 3308313000,
"status": "passed"
},
"line": 16,
"name": "Login to the application with below \"userName\" and \"password\"",
"match": {
"arguments": [
{
"val": "\"userName\"",
"offset": 36
},
{
"val": "\"password\"",
"offset": 51
}
],
"location": "LoginDefenition.java:21"
},
"rows": [
{
"cells": [
"userName",
"password"
]
},
{
"cells": [
"Qxz09gu",
"pSg8BpX1Gh20LNCTBGEiJA\u003d\u003d:W1lipUKQhdrZWs7yx/q0cBuggFxMtUhrK3HeWGsbQA0\u003d"
]
}
],
"keyword": "Given "
},
{
"result": {
"duration": 3771374800,
"status": "passed"
},
"line": 19,
"name": "Selecting the dealing company as \"FS_DE\"",
"match": {
"arguments": [
{
"val": "\"FS_DE\"",
"offset": 33
}
],
"location": "LoginDefenition.java:31"
},
"keyword": "When "
},
{
"result": {
"duration": 34198099900,
"status": "passed"
},
"line": 20,
"name": "Selecting an eligible transaction",
"match": {
"location": "LoginDefenition.java:37"
},
"keyword": "And "
}
],
"tags": [
{
"name": "@CFROF-19375"
},
{
"name": "@CFROF-19457"
}
]
}
],
"name": "",
"description": "",
"id": "",
"keyword": "Feature",
"uri": "file:src/test/resources/features/1.feature",
"tags": [
{
"name": "@CFROF-19375",
"type": "Tag",
"location": {
"line": 1,
"column": 1
}
}
]
}
]
Thanks in advance for your support :)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@rgnair , I think your info object (xrayresultimport.json) is not correct due to <build-tag>, <jira-fixed-version-id>, <build-url>.
Try to hardcode values for it first of all.
You should check Jenkins pipeline syntax in order to see how to pass variables; I *think* it should be something like the following, but I'm not 100% sure
${myVariable}
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
hi @Sergio Freire - Xblend i have used this endpoint to update the description filed but its giving me wrong results
curl -D- -u xxx:xxx -X PUT --data @C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json https://xxxxxxx/rest/api/2/issue/CFROF-19375
this is what i am getting
HTTP/1.1 405 Not Allowed
Date: Wed, 09 Sep 2020 09:42:34 GMT
Server: nginx/1.11.6
Content-Type: text/html
Content-Length: 173
<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>
My json data
{
"fields": {
"summary": "data",
"description": "data "
}
}
Any idea on how to update a jirs issue field(for me description)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
thanks @Sergio Freire - Xblend is there anyway i can update the custoem fileds in an issue key(for me description and summary needs to update)
i have curled this command and getting same result,any idea about how to get this or how can i update the values
>curl -D- -u xxx:xxx -X PUT --data @C:\CFS-Workspace\testautomation\jenkinsfiles\testautomation\xrayresultimport.json https://server/rest/raven/1.0/api/2/issue/CFROF-19375
HTTP/1.1 405 Not Allowed
Date: Wed, 09 Sep 2020 09:42:34 GMT
Server: nginx/1.11.6
Content-Type: text/html
Content-Length: 173
<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>
my json data
{
"fields": {
"summary": "Test Execution ",
"description": "Test Execution "
}
}
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@rgnair you should add the Content-Type header in your HTTP PUT request.
curl -v -u admin:admin -X PUT -H "Content-Type: application/json" --data @update_issue.json http://192.168.56.102/rest/api/2/issue/CALC-7877
This is my update_issue.json, as a mere example
{
"fields": {
"summary": "Test Execution for cucumber Execution",
"description": "example",
"customfield_11805" : [
"iOS"
],
"fixVersions" :
[
{
"name": "v3.0"
}
]
}
}
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
hi @Sergio Freire - Xblend great and thanks its worked :) but there are diffrenece in the custome field id's it seems.
errorMessages":[],"errors":{"fixVersions":"Version name 'v3.0' is not valid"}}* Closing connection 0
* schannel: shutting down SSL/TLS connection with atxxc.xxxxxx.net port 443
* schannel: clear security context handle
When i hit this with decsription only its worked :)...from where can i get the custome fileds ids to pass through this json file,any docs
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Yes, please consult ask your Jira admin to provide them, it can be seen in the Jira administration "custom fields" section.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Besides, the error you're having is because that project doesn't have a release named "v3.0". You need to adjust the request to your scenario.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.