How to add iterations when importing teste execution result via xray API

[JohnnyDoe]

I am trying to send test execution result with iterations:

{

   "testExecutionKey":"JIRA-0000",

   "tests":[

      {

         "testKey":"JIRA-0001",

         "status":"PASS",

         "start" : "2023-08-03T13:46:59.297",

          "finish" : "2023-08-03T13:46:59.301",

         "comment": "TEST COMMENT LMAO",

         "iterations":[ 

        {

                "name": "Iteration 1",

                "status": "PASSED",

              }

         ]

      }

   ]

}

I get error:

"

"error": "Error instantiating bean. Field(s) tests -> iterations -> name do not follow the correct format.""

When i literally copy paste example :

{

    "tests": [

        {

            "start" : "2021-08-30T11:47:35+01:00",

            "finish" : "2021-08-30T11:50:56+01:00",

            "comment" : "Successful execution",

            "status" : "PASSED",

            "evidence" : [

                {

                    "data": "iVBORw0KGgoAAAANSUhEUgAABkIAAAO9CAYAAADezXv6AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEn(...base64 file enconding)",

                    "filename": "image21.jpg",

                    "contentType": "image/jpeg"

                }

            ],

            "testInfo": {

                "summary": "Strong password validation",

                "type": "Manual",

                "projectKey": "STORE",

                "steps": [

                    {

                        "action": "Open the Change Password screen by selecting option \"My Profile > Password\"",

                        "data": "",

                        "result": ""

                    },

                    {

                        "action": "Fill the password fields with data",

                        "data": "Current Password: ${Password}\nNew Password: ${Password}\nConfirm New Password: ${Password}",

                        "result": "The new password is: ${Valid}\nError:\n${Message}"

                    }

                ]

            },

            "iterations": [

              {

                "name": "Iteration 1",

                "parameters": [

                    {

                        "name": "Password",

                        "value": "2635ftvu23v7t!09"

                    },

                    {

                        "name": "Valid",

                        "value": "Valid"

                    },

                    {

                        "name": "Message",

                        "value": ""

                    }

                ],

                "log": "Password changed successfully",

                "status": "PASSED",

                "steps": [

                    {

                        "actualResult": "",

                        "status": "PASSED"

                    },

                    {

                        "actualResult": "Password changed successfully",

                        "status": "PASSED"

                    }

                ]

              },

              {

                "name": "Iteration 2",

                "parameters": [

                    {

                        "name": "Password",

                        "value": "123123"

                    },

                    {

                        "name": "Valid",

                        "value": "Not Valid"

                    },

                    {

                        "name": "Message",

                        "value": "Password is too simple."

                    }

                ],

                "log": "Password validation check failed. Password too simple!",

                "status": "FAILED",

                "steps": [

                    {

                        "actualResult": "",

                        "status": "PASSED"

                    },

                    {

                        "actualResult": "Password too simple!",

                        "status": "FAILED"

                    }

                ]

              }

            ]

        }

    ]

}

i get the same error message.... what is correct format of iterations then?

Thanks for help

[Marc -Devoteam-]

Hi @JohnnyDoe 

If I look into importing Iterations, Iterations can only have and ID and STATUS object.

[JohnnyDoe]

Hi, where did u see it ? https://docs.getxray.app/display/XRAYCLOUD/Using+Xray+JSON+format+to+import+execution+results#UsingXrayJSONformattoimportexecutionresults-stepresult according to his it has soo much more stuff in it. It indeed works if i just add status however status is not reflected in actuall result on web

[Marc -Devoteam-]

Hi @JohnnyDoe 

You mentioned in tags you are running on Jira server, not Cloud.

The provided link points to the Xray Cloud API, not the Xray server/DC API.

See this link for the API on server/DC

[JohnnyDoe]

@Marc -Devoteam-  I checked it and there is no info about iterations

[JohnnyDoe]

@Marc -Devoteam- on top of that, given if i add dataset and add steps in UI, and then try to send it like:

{

   "testExecutionKey":"JIRA-0000",

   "tests":[

      {

         "testKey":"JIRA-0001",

         "status":"PASS",

         "start" : "2023-08-03T13:46:59.297",

          "finish" : "2023-08-03T13:46:59.301",

         "comment": "TEST COMMENT",

         "iterations":[{

              "status": "PASS"

            },

            {

              "status": "PASS"

            },{

              "status": "FAIL"

            }

         ]

      }

   ]

}

Only 1 out of N iterations ( based on number of data in data set ) is marked a "PASS" all others are ignored

[Marc -Devoteam-]

Hi @JohnnyDoe

Could you confirm you are on Jira Server or DC?

See the link I mentioned.

Look at the Test Run section

and in the scheme you can find Iterations

 

[JohnnyDoe]

@Marc -Devoteam-  I should be on server, but not sure ( i asked lot of people and nobody really knows.. ), as i menitoned, if i add multiple iterations only the first one gets picked up. I omit ID - as i dont know the ID since i run test from java and import results into test execution.

[Marc -Devoteam-]

Hi @JohnnyDoe 

For every import of a test run with multiple iterations you should be able to start with ID 1, for the second run you do the ID then should be 2.

[JohnnyDoe]

@Marc -Devoteam- hi,

{

   "testExecutionKey":"JIRA-0000",

   "tests":[

      {

         "testKey":"JIRA-0001",

         "status":"PASS",

         "start" : "2023-08-03T13:46:59.297",

          "finish" : "2023-08-03T13:46:59.301",

         "comment": "TEST COMMENT",

         "iterations":[{

               "id":1,

              "status": "PASS"

            },

            {

                "id":2,

              "status": "PASS"

            },{

                "id":3,

              "status": "FAIL"

            }

         ]

      }

   ]

}

Results in :

  "error": "Error instantiating bean. Field(s) tests -> iterations -> id do not follow the correct format."

[JohnnyDoe]

@Marc -Devoteam- Once again it seem the documentation is not valid..

[Marc -Devoteam-]

Hi @JohnnyDoe 

I would suggest to contact Xray in relation to your issue and the documentation.

They should then be able to explain you the way and might need to change there documentation.

I would highlight this when in contact with them.

[JohnnyDoe]

@Marc -Devoteam- on top of that your link suggests  to directly edit TEST RUN, i am running tests locally/on CI tools, so when the test finishes i only know test execution key and the test's key, not test runs.

[Marc -Devoteam-]

Hi @JohnnyDoe 

I have one more question.

The Tests created in Jira (I assume the Xray Test issuetype is used), what are the test type you created?

• Manual
• BDD
• Generic

In the documentation from Xray, I found the following: 

"It is also possible to import iteration results with parameters. Currently, this is only supported for manual tests."

To explain some more.

A Test Run occurs when a Test is scheduled to execute within a Test Execution. Therefore, each time you run a Test, you're, in fact, running a Test Run.

[JohnnyDoe]

@Marc -Devoteam- I am using manual type of test.  ( But i have cucumber test - so thats i want to have iterations and use manual type of test - each iteration represent specific run with different arguments. I have manually added data set to the xray test on UI.

I have managed to make it work however this rewrites all dataset values:

{

   "testExecutionKey":"JIRA-",

   "tests":[

      {

         "testKey":"JIRA-1",

         "status":"PASS",

         "start" : "2023-08-03T13:46:59.297",

          "finish" : "2023-08-03T13:46:59.301",

         "comment": "TEST COMMENT",

            "iterations": [

              {

                "parameters": [

                    {

                        "name": "Iteration",

                        "value": "Iteration 1"

                    }

                ],

                "status": "PASS"

              },

              {

                "parameters": [

                    {

                        "name": "Iteration",

                        "value": "Iteration 2"

                    }

                ],

                "status": "FAIL"

              },

              {

                "parameters": [

                    {

                        "name": "Iteration",

                        "value": "Iteration 3"

                    }

                ],

                "status": "PASS"

              }

            ]

      }

   ]

}

When i omit "parameters" from iteration i get error that there are duplicated iterations. So i cant just do

            "iterations": [

              {

                "status": "PASS"

              },

              {

                "status": "PASS"

              },

              {

                "status": "PASS"

              }

            ]

But thats exactly what i need to do, since when i add parameters to the iteration the one i set in dataseton UI gets overwritten.

Is there any way to achieve this without overwritting data set?