PHPUnit fails, Pipelines doesn't find error in test report

Kim Metso September 25, 2020

I have a pipeline that runs PHPUnit. One of the tests fail, and PHPUnit writes a test report:

Screenshot 2020-09-25 at 16.47.12.png

 

The test report contains a record of the failure:

Screenshot 2020-09-25 at 16.48.43.png

 

Yet Pipelines doesn't find the error:

Screenshot 2020-09-25 at 16.49.05.png

 

The tests are consequently considered a success, which means the Tests tab isn't visible. The pipeline itself correctly fails because of the exit code from phpunit.

The expected result is that I should see the Tests tab, and within the tab I should see the failed test.

Is this a bug in Pipelines or am I doing something wrong?

3 answers

1 vote
Elisa de Llano September 9, 2021

I have the same issue, did someone ever find a culprit or a solution?

Elisa de Llano September 10, 2021

Mixing tests with a data provider and tests without is what causes this.

Like ktomk likes this
0 votes
Oleksandr Kyrdan
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 25, 2020

Hi @Kim Metso ,

Thank you for your question!

Let's try command:

vendor/bin/phpunit --log-junit=test-reports/report.xml

 more details provided in the Test reporting in Pipelines guide.

Kim Metso September 25, 2020

Thank you for your suggestion @Oleksandr Kyrdan. I tried this, but the result was the same as in my original post. The test report parser only finds 129 tests.

I'm not sure your suggested command does anything differently from my original one? My original command was from the page you are linking to.

Looking at the issues for PHPUnit (the ones concerning junit specifically), it seems the report format isn't very standardized. I tried updating PHPUnit to 8.5.8 but the resulting report is identical.

ktomk
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 26, 2020

Can you clarify if the output xml contains 129 or 132 test results?

Can you identify the 3 tests that are not reported by the test-reporter on Bitbucket? (Maybe not being reported as part of a test-suite or whatever this could be).

Phpunit invocation looks good to me: xml log file is created and found by test-reporter on bitbucket already in the OP one.

Kim Metso September 26, 2020

@ktomk The XML file does contain 132 test results.

Here is the basic structure of the file (with a lot of stuff omitted for brevity):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="" tests="132" assertions="491" errors="0" warnings="0" failures="1" skipped="0" time="0.823959">
<!-- ... -->
<testsuite name="Feature" tests="109" assertions="468" errors="0" warnings="0" failures="1" skipped="0" time="0.645507">
<!-- ... -->
<testsuite name="Tests\Feature\ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" tests="71" assertions="304" errors="0" warnings="0" failures="1" skipped="0" time="0.403150">
<testsuite name="Tests\Feature\ObjectPageFieldsTest::it_prints_the_correct_value_for_the_field" tests="35" assertions="268" errors="0" warnings="0" failures="0" skipped="0" time="0.267753">
<testcase name="it_prints_the_correct_value_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="7" time="0.008839"/>
<testcase name="it_prints_the_correct_value_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="8" time="0.007462"/>
<!-- ... -->
</testsuite>
<testsuite name="Tests\Feature\ObjectPageFieldsTest::it_has_a_label_for_the_field" tests="35" assertions="35" errors="0" warnings="0" failures="0" skipped="0" time="0.131400">
<testcase name="it_has_a_label_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.004041"/>
<testcase name="it_has_a_label_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.003846"/>
<!-- ... -->
</testsuite>
<testcase name="test_exists_for_all_fields" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="109" assertions="1" time="0.003998">
<failure type="PHPUnit\Framework\ExpectationFailedException">
<!-- PHPUnit assertion failure output here -->
</failure>
</testcase>
</testsuite>
</testsuite>
</testsuite>
</testsuites>

I have no idea what the structure of the file should be like, so I can't really say what could be wrong. The nested <testsuite> elements seem kind off odd, but on the other hand the report parser seems to be able to find most tests so I assume that's valid.

Some of the tests use a PHPUnit dataProvider, which seems to make those tests a testsuite by themselves. It makes sense, since the functions are in fact run multiple times but with different data. Each invocation is being output as a <testcase>.

The failing test, "test_exists_for_all_fields", does not use a dataProvider and is being output as a <testcase>, as a sibling to the <testsuite>s using dataProviders.

0 votes
Kim Metso September 25, 2020

A clue might be in that the PHPUnit output in the first screenshot says 132 tests, while the total count in the last screenshot says 129 tests. So I guess that failure ins't found/recognized. But why?

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events