Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,363,489
Community Members
 
Community Events
168
Community Groups

PHPUnit fails, Pipelines doesn't find error in test report

I have a pipeline that runs PHPUnit. One of the tests fail, and PHPUnit writes a test report:

Screenshot 2020-09-25 at 16.47.12.png

 

The test report contains a record of the failure:

Screenshot 2020-09-25 at 16.48.43.png

 

Yet Pipelines doesn't find the error:

Screenshot 2020-09-25 at 16.49.05.png

 

The tests are consequently considered a success, which means the Tests tab isn't visible. The pipeline itself correctly fails because of the exit code from phpunit.

The expected result is that I should see the Tests tab, and within the tab I should see the failed test.

Is this a bug in Pipelines or am I doing something wrong?

3 answers

I have the same issue, did someone ever find a culprit or a solution?

Mixing tests with a data provider and tests without is what causes this.

Like ktomk likes this
0 votes

Hi @slackernrrd ,

Thank you for your question!

Let's try command:

vendor/bin/phpunit --log-junit=test-reports/report.xml

 more details provided in the Test reporting in Pipelines guide.

Thank you for your suggestion @Oleksandr Kyrdan. I tried this, but the result was the same as in my original post. The test report parser only finds 129 tests.

I'm not sure your suggested command does anything differently from my original one? My original command was from the page you are linking to.

Looking at the issues for PHPUnit (the ones concerning junit specifically), it seems the report format isn't very standardized. I tried updating PHPUnit to 8.5.8 but the resulting report is identical.

ktomk Rising Star Sep 26, 2020

Can you clarify if the output xml contains 129 or 132 test results?

Can you identify the 3 tests that are not reported by the test-reporter on Bitbucket? (Maybe not being reported as part of a test-suite or whatever this could be).

Phpunit invocation looks good to me: xml log file is created and found by test-reporter on bitbucket already in the OP one.

@ktomk The XML file does contain 132 test results.

Here is the basic structure of the file (with a lot of stuff omitted for brevity):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="" tests="132" assertions="491" errors="0" warnings="0" failures="1" skipped="0" time="0.823959">
<!-- ... -->
<testsuite name="Feature" tests="109" assertions="468" errors="0" warnings="0" failures="1" skipped="0" time="0.645507">
<!-- ... -->
<testsuite name="Tests\Feature\ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" tests="71" assertions="304" errors="0" warnings="0" failures="1" skipped="0" time="0.403150">
<testsuite name="Tests\Feature\ObjectPageFieldsTest::it_prints_the_correct_value_for_the_field" tests="35" assertions="268" errors="0" warnings="0" failures="0" skipped="0" time="0.267753">
<testcase name="it_prints_the_correct_value_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="7" time="0.008839"/>
<testcase name="it_prints_the_correct_value_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="8" time="0.007462"/>
<!-- ... -->
</testsuite>
<testsuite name="Tests\Feature\ObjectPageFieldsTest::it_has_a_label_for_the_field" tests="35" assertions="35" errors="0" warnings="0" failures="0" skipped="0" time="0.131400">
<testcase name="it_has_a_label_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.004041"/>
<testcase name="it_has_a_label_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.003846"/>
<!-- ... -->
</testsuite>
<testcase name="test_exists_for_all_fields" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="109" assertions="1" time="0.003998">
<failure type="PHPUnit\Framework\ExpectationFailedException">
<!-- PHPUnit assertion failure output here -->
</failure>
</testcase>
</testsuite>
</testsuite>
</testsuite>
</testsuites>

I have no idea what the structure of the file should be like, so I can't really say what could be wrong. The nested <testsuite> elements seem kind off odd, but on the other hand the report parser seems to be able to find most tests so I assume that's valid.

Some of the tests use a PHPUnit dataProvider, which seems to make those tests a testsuite by themselves. It makes sense, since the functions are in fact run multiple times but with different data. Each invocation is being output as a <testcase>.

The failing test, "test_exists_for_all_fields", does not use a dataProvider and is being output as a <testcase>, as a sibling to the <testsuite>s using dataProviders.

A clue might be in that the PHPUnit output in the first screenshot says 132 tests, while the total count in the last screenshot says 129 tests. So I guess that failure ins't found/recognized. But why?

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events