Issues/Bug With Importing Test Execution JSON Results with New Tests

Question

My Json input file looks like this for testing minimal,

{
"info": {
"project" : "project"
  },
    "tests" : [
      {
"status": "TODO",
        "testInfoBean": {
"summary": "Automation Test",
          "projectKey": "project",
          "genericTestDefinition": "class-api.method-enablesomething"
        }
      }
    ]
}

What is expected, following this formatting guide and the bottom of this page, is that by using "genericTestDefinition" a new generic test should be created if a matching case is not found with an equal genericTestDefinition.

"genericTestDefinition": "class-api.method-enablesomething"

What is observed is that, a manual test is created in our Jira/Xray instance.

Additionally, when I try to use the manualTestDefinition field, I can not create a successful case with the required step field. I only receive back an error stating:  {"error":"Error instantiating bean. Field(s) tests -\u003e testInfoBean -\u003e manualTestDefinition do not follow the correct format.

where, again same reference, my JSON input is:

{ 
"info": {
"project": "project" 
  }, 
"tests": [ 
    { 
"status": "TODO", 
"testInfoBean": { 
"summary": "CreatedTestManual", 
"projectKey": "project", 
"manualTestDefinition": [ 
            { "action": "step1" } 
         ] 
      } 
    } 
  ] 
}

Brian Grecho · Answer

Thank you so much Rogerio for the reply!

ok so, i was using this same reference,

starting at ID XrayNewTest:

testInfoBean: {
  oneOf: [
    {
      $ref: "/TestCreateInfoManual"
    },
    {
      $ref: "/TestCreateInfoCucumber"
    },
    {
      $ref: "/TestCreateInfoGeneric"
    }
  ]
},

I guess i am not understanding this formatting guide properly:

const TestCreateInfoManual = {
    id: "/TestCreateInfoManual",
    type: "object",
    properties: {
       summary: {
           type: "string"
        },
        projectKey: {
           type: "string"
        },
        manualTestDefinition: {
           type: "array",
           items: {
               $ref: "/ManualStep"
           }
      },
      requirementKey: {
          type: "string"
      },
      labels: {
          type: "array",
          items: {
              type: "string"
          }
       }
    },
    required: ["summary", "projectKey", "manualTestDefinition"],
    additionalProperties: false
};

and then ManualStep is:

const ManualStep = {
    id: "/ManualStep",
    type: "object",
    properties: {
        action: {
            type: "string"
        },
        data: {
            type: "string"
        },
        result: {
            type: "string"
        }
    },
    required: ["action"],
    additionalProperties: false
};

and as posted above my formatting as i understood was:

{ 
"info": {
"project": "project" 
  }, 
"tests": [ 
    { 
"status": "TODO", 
"testInfoBean": { 
"summary": "CreatedTestManual", 
"projectKey": "project", 
"manualTestDefinition": [ 
            { "action": "step1" } 
         ] 
      } 
    } 
  ] 
}

Can you help me see what I'm missing here?

thank you so much!

Rogério Paiva - Xray Xporter · Answer

Hi @Brian Grecho

In Xray, each reporting format has its own REST API and Test Type associated.

For instance, Xray JSON format is to be used with Manual Test Types.

The information you found at the bottom of the page, is about the JUnit format that uses an XML reporting format and is of Generic Test Type.

So if you want to use Xray JSON, it will be a Manual Test Type, if you want to have a Generic Test Type, it is necessary to use one of the formats that work as Generic, like JUnit, NUnit, xUnit, TestNG, Robot.

About the second issue, your JSON is not following the correct format, please check the Xray JSON format here (https://docs.getxray.app/display/XRAYCLOUD/Import+Execution+Results#ImportExecutionResults-XrayJSONformat).

If you need additional help, please contact the Xray Support (http://xraysupport.xpand-it.com ) so that we can request additional information and help further.

Kind regards,
Rogério Paiva [Xray Support Team]