Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Wrong Test count on Maven surefire Test reporting

Hello Team,

 when I execute Tests in the pipeline multiple XML gets scanned in the target output directory. Pipeline scan all the XML show test count in multiples of 3.

How can we avoid this wrong Test count representation?

Logs:

Searching for test report files in directories named [test-reports, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/Suite/Test.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/TEST-TestSuite.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/junitreports/TEST-com.practice.SampleTest.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/testng-results.xml
Finished scanning for test reports. Found 4 test report files.
Merged test suites, total number tests is 6, with 0 failures and 0 errors.

1 answer

0 votes

Hi @Prashant Raju Navkudkar,

I have highlighted some parts of the log you posted here:

Searching for test report files in directories named [test-reports, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/Suite/Test.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/TEST-TestSuite.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/junitreports/TEST-com.practice.SampleTest.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/testng-results.xml
Finished scanning for test reports. Found 4 test report files.
Merged test suites, total number tests is 6, with 0 failures and 0 errors.

Based on the highlighted output, it looks that the scanner found 2 tests in Test.xml, 2 tests in TEST-TestSuite.xml and 2 tests in TEST-com.practice.SampleTest.xml, which makes for 6 tests in total.

Could you please explain why the test count is wrong, so we can better help you?

Kind regards,
Theodora

Here I have executed 2 Tests that generate three XML results, so here by default pipeline collects Test execution data from all the XMLs which results in 2X3(XML) = 6 Tests as the total count. Ideally, Test count is supposed to be 2 only.  

There should be a way to specify a particular XML in the pieline.

Hi Prashant,

Thank you for your reply and explanation.

Just to give you some context here on why this is occurring: the test report file scanner will begin searching from the base directory of your build: /opt/atlassian/pipelines/agent/build and it will look for reports in any of the following locations

./**/surefire-reports/**/*.xml
./**/failsafe-reports/**/*.xml
./**/test-results/**/*.xml
./**/test-reports/**/*.xml

At the moment, I'm afraid that it is not possible to specify a specific directory or XML file only.

However, we have a feature request for this in our issue tracker:

I would suggest that you vote for and watch this issue.

Our development team has also asked customers who are interested in this to explain their use case for configuring it explicitly in the yaml file. Please feel free to let me know if you would like me to post details of your use case in the feature request, or if you prefer to leave a comment there yourself.

Just to understand your use case better, I'd like to ask if it's expected behavior to have 3 XML files generated for the same tests? Is there a possibility to configure maven/surefire to generate only one, or generate some of the reports in a location other than the ones I mentioned above?

Kind regards,
Theodora

Like Prashant Raju Navkudkar likes this

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Jira

Admins, notify your Jira instance of system-wide changes with the new admin announcement banner

Hi All! We’re excited to share the launch of an announcement banner that lets Jira site administrators communicate directly to their users across their  Jira Cloud instance.  ...

545 views 15 18
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you