Help displaying custom test results

[Mark]

I have a custom build, that does not use a typical language that has Xunit tests, so I'm not using any particular testing setup. I do however have tests, I can format these in any format that is needed but I need to know what the format is.


I've tried the Xunit, JUnit, and another one that I'm not sure what it was now. In any case, at the end of my build the bitbucket pipeline finds the files, does _something_ to merge them but then finds no tests. I'm assuming that because it's not finding what it wants to find, it is not showing the "Reports" link or the tab for "Tests".

Since this is implicit behavior of the bitbucket pipelines, I have no insight into what might be going wrong here.

Is there some way to display more debugging information for this test merge process?

Is there a specific example of what test result format works for these?

Here is a snippet from the bottom of my log from my pipeline:

Searching for test report files in directories named [test-results, failsafe-reports, test-reports, surefire-reports] down to a depth of 4
Found matching test report file /opt/atlassian/pipelines/agent/build/test-results/all-test-results.xml
Found matching test report file /opt/atlassian/pipelines/agent/build/test-results/all-test-results2.xml
Found matching test report file /opt/atlassian/pipelines/agent/build/test-results/all-test-results3.xml
Finished scanning for test reports. Found 3 test report files.Merged test suites, total number tests is 0, with 0 failures and 0 errors.


[Oleksandr Kyrdan · Accepted solution]

Hi @Mark ,

Thank you for your question!

Note: the Tests tab will only show if you have failed tests to review.

Also, make sure that you provide the right path to tests reports (test-reports/report.xml).
Example for pytest:

script:
  - pytest test/test.py --junitxml=test-reports/report.xml

More details you can find in the Test reporting in Pipelines guide.

[Mark]

So, I have a custom build. I'm not using a standard testing set up, but I am trying to figure out the format so I can format my test results so I can see them in the reports.

I've mocked up some results and put them in a /test-results directory. The system seems to see these as I see 

Finished scanning for test reports. Found 3 test report files.

However, it must not like what it sees because it always says:

Merged test suites, total number tests is 0, with 0 failures and 0 errors.

In my mocked test results I definitely have some "failures".

[Mark]

So, in the end it was the formats I was using. One of mine was close, but was apparently an older format of the junitxml like in your example.

I tried another results file with the format put out by pytest and it it was able to parse it. Thank you for your help!

[Zuberu Abdulrahman]

Hi I am having similar issue. We are migrating of testing to bitbucket, we have been running it using robot framework. We are generating 3 files: 

Output: /opt/robotframework/reports/output.xml

Log: /opt/robotframework/reports/log.html

Report: /opt/robotframework/reports/report.html

But the piple is able to find only the xml file which I understand due to the format of the other files but it was not able to upload those files for me to view them, any idea why?

Searching for test report files in directories named [test-results, failsafe-reports, test-reports, surefire-reports] down to a depth of 4

Found matching test report file /opt/atlassian/pipelines/agent/build/sisko/reports/test-reports/output.xml

Finished scanning for test reports. Found 1 test report files.Merged test suites, total number tests is 0, with 0 failures and 0 errors.

Also the xml file only contains the data and the whole presentation is in those html files. Any idea how we can solve this issue?

[Mark]

I just generated some test results using pytest. I put those in the results directory to ensure they were in the right place, formatted correctly, and that they would display.

Once I had something that was working, I was able to compare what was working with what my tests were generating and I found differences. 

I think the test results are rather finicky on the exact formats. What I found that works is a format that looks like the following:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite errors="0" failures="0" hostname="none" name="somename.asm" skipped="0" tests="3" time="0" timestamp="2020-05-25T00:00:00.000000">
<testcase classname="SomeClass" file="somefile.asm" line="1" name="Some kind of name" time="0"></testcase>
<testcase classname="SomeClass" file="somefile.asm" line="2" name="Some kind of name" time="0"></testcase>
<testcase classname="SomeClass" file="somefile.asm" line="3" name="Some kind of name" time="0"></testcase>
</testsuite>
<testsuite errors="0" failures="1" hostname="none" name="someothername.asm" skipped="0" tests="3" time="0" timestamp="2020-05-25T00:00:00.000000">
<testcase classname="SomeOtherClass" file="someotherfile.asm" line="1" name="Some other kind of name" time="0"></testcase>
<testcase classname="SomeOtherClass" file="someotherfile.asm" line="2" name="Some other kind of name" time="0">
<failure message="This Failed">
...some details about failure...
...some details about failure...
...some details about failure...
...some details about failure...
...some details about failure...
</failure>
</testcase>
</testsuite>
</testsuites>