Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

PHPUnit fails, Pipelines doesn't find error in test report

I have a pipeline that runs PHPUnit. One of the tests fail, and PHPUnit writes a test report:

Screenshot 2020-09-25 at 16.47.12.png

 

The test report contains a record of the failure:

Screenshot 2020-09-25 at 16.48.43.png

 

Yet Pipelines doesn't find the error:

Screenshot 2020-09-25 at 16.49.05.png

 

The tests are consequently considered a success, which means the Tests tab isn't visible. The pipeline itself correctly fails because of the exit code from phpunit.

The expected result is that I should see the Tests tab, and within the tab I should see the failed test.

Is this a bug in Pipelines or am I doing something wrong?

2 answers

A clue might be in that the PHPUnit output in the first screenshot says 132 tests, while the total count in the last screenshot says 129 tests. So I guess that failure ins't found/recognized. But why?

0 votes

Hi @slackernrrd ,

Thank you for your question!

Let's try command:

vendor/bin/phpunit --log-junit=test-reports/report.xml

 more details provided in the Test reporting in Pipelines guide.

Thank you for your suggestion @Oleksandr Kyrdan. I tried this, but the result was the same as in my original post. The test report parser only finds 129 tests.

I'm not sure your suggested command does anything differently from my original one? My original command was from the page you are linking to.

Looking at the issues for PHPUnit (the ones concerning junit specifically), it seems the report format isn't very standardized. I tried updating PHPUnit to 8.5.8 but the resulting report is identical.

Can you clarify if the output xml contains 129 or 132 test results?

Can you identify the 3 tests that are not reported by the test-reporter on Bitbucket? (Maybe not being reported as part of a test-suite or whatever this could be).

Phpunit invocation looks good to me: xml log file is created and found by test-reporter on bitbucket already in the OP one.

@ktomk The XML file does contain 132 test results.

Here is the basic structure of the file (with a lot of stuff omitted for brevity):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="" tests="132" assertions="491" errors="0" warnings="0" failures="1" skipped="0" time="0.823959">
<!-- ... -->
<testsuite name="Feature" tests="109" assertions="468" errors="0" warnings="0" failures="1" skipped="0" time="0.645507">
<!-- ... -->
<testsuite name="Tests\Feature\ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" tests="71" assertions="304" errors="0" warnings="0" failures="1" skipped="0" time="0.403150">
<testsuite name="Tests\Feature\ObjectPageFieldsTest::it_prints_the_correct_value_for_the_field" tests="35" assertions="268" errors="0" warnings="0" failures="0" skipped="0" time="0.267753">
<testcase name="it_prints_the_correct_value_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="7" time="0.008839"/>
<testcase name="it_prints_the_correct_value_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="8" time="0.007462"/>
<!-- ... -->
</testsuite>
<testsuite name="Tests\Feature\ObjectPageFieldsTest::it_has_a_label_for_the_field" tests="35" assertions="35" errors="0" warnings="0" failures="0" skipped="0" time="0.131400">
<testcase name="it_has_a_label_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.004041"/>
<testcase name="it_has_a_label_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.003846"/>
<!-- ... -->
</testsuite>
<testcase name="test_exists_for_all_fields" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="109" assertions="1" time="0.003998">
<failure type="PHPUnit\Framework\ExpectationFailedException">
<!-- PHPUnit assertion failure output here -->
</failure>
</testcase>
</testsuite>
</testsuite>
</testsuite>
</testsuites>

I have no idea what the structure of the file should be like, so I can't really say what could be wrong. The nested <testsuite> elements seem kind off odd, but on the other hand the report parser seems to be able to find most tests so I assume that's valid.

Some of the tests use a PHPUnit dataProvider, which seems to make those tests a testsuite by themselves. It makes sense, since the functions are in fact run multiple times but with different data. Each invocation is being output as a <testcase>.

The failing test, "test_exists_for_all_fields", does not use a dataProvider and is being output as a <testcase>, as a sibling to the <testsuite>s using dataProviders.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket Pipelines

What We Learned When We Researched Open Source Vulnerabilities in 7 Popular Coding Languages

...hey are a part of us, shaping how we interact with the world around us. The same holds true for programming languages when we think about how different kinds of vulnerabilities raise their heads in t...

1,323 views 0 3
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you