Wrong Test count on Maven surefire Test reporting

Question

Hello Team,

when I execute Tests in the pipeline multiple XML gets scanned in the target output directory. Pipeline scan all the XML show test count in multiples of 3.

How can we avoid this wrong Test count representation?

Logs:

Searching for test report files in directories named [test-reports, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/Suite/Test.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/TEST-TestSuite.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/junitreports/TEST-com.practice.SampleTest.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/testng-results.xml
Finished scanning for test reports. Found 4 test report files.
Merged test suites, total number tests is 6, with 0 failures and 0 errors.

Theodora Boudale · Answer

Hi Prashant,

Thank you for your reply and explanation.

Just to give you some context here on why this is occurring: the test report file scanner will begin searching from the base directory of your build: /opt/atlassian/pipelines/agent/build and it will look for reports in any of the following locations

./**/surefire-reports/**/*.xml
./**/failsafe-reports/**/*.xml
./**/test-results/**/*.xml
./**/test-reports/**/*.xml

At the moment, I'm afraid that it is not possible to specify a specific directory or XML file only.

However, we have a feature request for this in our issue tracker:

* https://jira.atlassian.com/browse/BCLOUD-15419

I would suggest that you vote for and watch this issue.

Our development team has also asked customers who are interested in this to explain their use case for configuring it explicitly in the yaml file. Please feel free to let me know if you would like me to post details of your use case in the feature request, or if you prefer to leave a comment there yourself.

Just to understand your use case better, I'd like to ask if it's expected behavior to have 3 XML files generated for the same tests? Is there a possibility to configure maven/surefire to generate only one, or generate some of the reports in a location other than the ones I mentioned above?

Kind regards,
Theodora

Prashant Raju Navkudkar · Answer

Here I have executed 2 Tests that generate three XML results, so here by default pipeline collects Test execution data from all the XMLs which results in 2X3(XML) = 6 Tests as the total count. Ideally, Test count is supposed to be 2 only.

There should be a way to specify a particular XML in the pieline.

Theodora Boudale · Answer

Hi @Prashant Raju Navkudkar,

I have highlighted some parts of the log you posted here:

Searching for test report files in directories named [test-reports, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/Suite/Test.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/TEST-TestSuite.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/junitreports/TEST-com.practice.SampleTest.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/testng-results.xml
Finished scanning for test reports. Found 4 test report files.
Merged test suites, total number tests is 6, with 0 failures and 0 errors.

Based on the highlighted output, it looks that the scanner found 2 tests in Test.xml, 2 tests in TEST-TestSuite.xml and 2 tests in TEST-com.practice.SampleTest.xml, which makes for 6 tests in total.

Could you please explain why the test count is wrong, so we can better help you?

Kind regards,
Theodora