Importing test execution results with new test cases using Xray Json REST API

[Shawn Hesch]

I am attempting to import a list of test case execution results to a JIRA project using the Xray REST API.

Using the Xray JSON format I have successfully uploaded execution results for existing tests. Unfortunately I have not been able to upload results for new tests. 

The JSON used for the upload attempt is: 

{

    "testExecutionKey" : "NXTP-40",

    "info" : {

        "summary" : "JSON Test Execution"

    },

    "tests" : [

        {

            "status" : "PASSED",

            "testInfoBean" :

            {

                "summary" : "JSONCreatedTest",

                "projectKey" : "NXTP",

                "genericTestDefinition" : "tes.definition"

            }

        }

     ]

}

When this JSON is uploaded using the REST API or using the test execution importer tool in JIRA, an error is returned stating there are no valid tests

{"error":"There are no valid tests imported"}

Based on the JSON format outlined in the documentation I am unable to find which fields are incorrect or missing.

Any help pointing to how the JSON needs to be formatted to create a new test via test execution upload would be greatly appreciated.

Regards,

Shawn

[Rogério Paiva - Xray Xporter · Accepted solution]

Hi,

This feature has been released in Xray version 5.1.0.

Please check the documentation here: https://docs.getxray.app/display/XRAY/Import+Execution+Results#ImportExecutionResults-XrayJSONformat

Thank you.

Kind regards,
Rogério Paiva [Xray Support Team]

[Shawn Hesch · Accepted solution]

After discussing this issue with Xray support I was informed that isn't currently possible using the Xray JSON format and that one of the XML formats would have to be used to accomplish my goals.

[Kartheeswaran Ramasamy]

I am facing the same problem, I have framework net6.0 , and trying to generate Specflow test result in xml but I am failing in it, its not generating for me, could you give some idea how to create it

[rbalamu2]

Hi Team,

Importing test execution results with new test cases/updating test cases using Xray CSV is possible? Kindly help. We are looking for a solution using csv.

Thanks,

Rahini

[Michal Chmielewski]

Hi. Is there any information about when this issue will be fixed? The project I am working on relies heavily on the importing of test results, and we cannot move forward otherwise. It would be very helpful to be able to upload through XRay Json format. 

Thanx :) 

[Eric Olson]

Hey Michal. I don't know why they did it this way, but they ended up forking this conversation into a new feature request on their jira board that they never linked here: https://jira.getxray.app/browse/XRAY-6392

They just recently notified me that the feature was released in 5.1.0. I haven't had a chance to try it yet to confirm that it works (waiting on my Jira admins to update the plugin), but theoretically it's out there in the wild!

[Eric Olson]

This is a serious shortcoming, and it's very confusing from a design perspective. The functionality (creating new tests automatically) already exists with some of the XML formats (TestNG, for example), but it doesn't work with your own proprietary XRay JSON format.

The integration I'm building becomes orders of magnitude more complex as a result. Consider, instead of:

1. Create execution payload
2. For each test, create test run payload, using either known Jira key or some standardized signature
3. Send to XRay API

It becomes:

1. Create execution payload
2. Query Jira API for my project, pulling down all issues with the appropriate issueType
3. Iterate over all test runs, doing some custom matching between a signature of my choosing and whatever lives on the Jira tickets returned from the query.
   
   1. If a signature matches, modify the appropriate test run payload to now hold the testKey pulled from the Jira ticket
   2. If it doesn't match anything in the query, store it off to the side for the moment
4. Now iterate over all of the unmatched runs, calling the Jira API to create a new issue (1 by 1).
5. For each created ticket, modify the associated test run payload to have the new testKey
6. Send to XRay API

Our company is large with a wide diversity of team processes. Some of our teams are fine manually tagging all of their tests - especially those who still have a mix of manual and automated testing, but a good chunk of our projects have shifted to a fully-automated paradigm with hundreds or thousands of tests, and having to retroactively apply thousands of pieces of metadata to their codebase will be untenable.

Please fix this, @José Domingues [Xray] and thanks for leading the way on this request, @Shawn Hesch 

[Eric Olson]

Oh, as an addendum, this problem is kind of hinted at via this section of the doc: https://docs.getxray.app/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-HowresultsaremappedtoTestentities

It took me a really long time to find this section, however, and even when I did find it, it didn't say anything about the XRay Json format, so I kept struggling with it until eventually I searched and found this post. It probably should be made more visible, and it should enumerate which exact formats can use the feature and which can't, rather than just giving a couple examples of each.

[José Domingues [Xray]]

Hi @Shawn Hesch ,

When importing execution results using the Xray JSON result format in a Continuous Integration environment, you can specify which Test Execution issue to import the results on using the testExecutionKey property. Alternatively, you can create a new Test Execution for the execution results and specify the Test Execution issue fields in the info object.

We suggest you check our documentation and adapt your report to the example you can see here:  https://confluence.xpand-it.com/display/XRAYCLOUD/Import+Execution+Results+-+REST#ImportExecutionResultsREST-XrayJSONresults.

If you have any more questions, please submit a support request in our Service Desk. 

Best Regards,

Team Xray

[Shawn Hesch]

Thank you for the information @José Domingues [Xray],

In this case I was attempting to create a new test case, not test execution.

After discussing this issue with Xray support I was informed that isn't currently possible using the Xray JSON format and that one of the XML formats would have to be used to accomplish my goals.

Regards,

Shawn