Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,462,584
Community Members
 
Community Events
176
Community Groups

Wrong Test count on Maven surefire Test reporting

Hello Team,

 when I execute Tests in the pipeline multiple XML gets scanned in the target output directory. Pipeline scan all the XML show test count in multiples of 3.

How can we avoid this wrong Test count representation?

Logs:

Searching for test report files in directories named [test-reports, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/Suite/Test.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/TEST-TestSuite.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/junitreports/TEST-com.practice.SampleTest.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/testng-results.xml
Finished scanning for test reports. Found 4 test report files.
Merged test suites, total number tests is 6, with 0 failures and 0 errors.

1 answer

0 votes

Hi @Prashant Raju Navkudkar,

I have highlighted some parts of the log you posted here:

Searching for test report files in directories named [test-reports, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/Suite/Test.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/TEST-TestSuite.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/junitreports/TEST-com.practice.SampleTest.xml
Located test suite containing 2 tests, with 0 failures and 0 errors.
Found matching test report file /tmp/2dc7cb44-ab4b-5c0f-b7c7-4d3d07d461a9/build/target/surefire-reports/testng-results.xml
Finished scanning for test reports. Found 4 test report files.
Merged test suites, total number tests is 6, with 0 failures and 0 errors.

Based on the highlighted output, it looks that the scanner found 2 tests in Test.xml, 2 tests in TEST-TestSuite.xml and 2 tests in TEST-com.practice.SampleTest.xml, which makes for 6 tests in total.

Could you please explain why the test count is wrong, so we can better help you?

Kind regards,
Theodora

Here I have executed 2 Tests that generate three XML results, so here by default pipeline collects Test execution data from all the XMLs which results in 2X3(XML) = 6 Tests as the total count. Ideally, Test count is supposed to be 2 only.  

There should be a way to specify a particular XML in the pieline.

Hi Prashant,

Thank you for your reply and explanation.

Just to give you some context here on why this is occurring: the test report file scanner will begin searching from the base directory of your build: /opt/atlassian/pipelines/agent/build and it will look for reports in any of the following locations

./**/surefire-reports/**/*.xml
./**/failsafe-reports/**/*.xml
./**/test-results/**/*.xml
./**/test-reports/**/*.xml

At the moment, I'm afraid that it is not possible to specify a specific directory or XML file only.

However, we have a feature request for this in our issue tracker:

I would suggest that you vote for and watch this issue.

Our development team has also asked customers who are interested in this to explain their use case for configuring it explicitly in the yaml file. Please feel free to let me know if you would like me to post details of your use case in the feature request, or if you prefer to leave a comment there yourself.

Just to understand your use case better, I'd like to ask if it's expected behavior to have 3 XML files generated for the same tests? Is there a possibility to configure maven/surefire to generate only one, or generate some of the reports in a location other than the ones I mentioned above?

Kind regards,
Theodora

Like Prashant Raju Navkudkar likes this

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events