Tests are failing but pipeline step still passes ??

[Adam Brunt]

Hi all,

We have a situation where a pipeline step is passing even though Bitbucket detects failing tests for that step.

The step summary says "1 / 1894 tests failed", the console log has a "Tests" tab on it which shows the failing test and the build teardown log shows "Merged test suites, total number tests is 1894, with 1 failures and 0 errors." but the step is still green.

If it makes a difference we write our test results to a file called test-reports/results.xml. though I notice the teardown log says "Searching for test report files in directories named [test-results, failsafe-reports, test-reports, TestResults, surefire-reports]". Does anyone know if the folder name makes a difference ?

Does anyone have any idea what our problem might be ? Or is it a bug with Bitbucket Pipelines ?

[Patrik S]

Hey @Adam Brunt ,

and welcome to the Community!

The way that Bitbucket Pipelines defines whether a step was successful or not is based on the exit code returned by the commands you're executing in your script.

An exit code of zero is interpret as the command being successful, and pipelines follow to the next command, which then returns it's own exit code as well, and so on.

If all commands in a step return exit code zero, the step will be considered successful. On the other hand, if any command returns a non-zero exit code, Pipelines will stop that step immediately and consider it as a failure.

That being said, and bringing this to your scenario of tests results, the test results contents itself doesn't impact whether the pipeline will consider the step as failed on not. What actually matter is what exit code your test framework has returned once it complete its execution. If your test suite identified failed tests but still returned a zero exit code, pipelines will understand that command succeeded.

I'd suggest including the following command right after your test execution command to print the exit code:

- echo $?

This command outputs the exit code of the last command executed in the script. If it's zero, that would explain why the pipelines considered it successfully.

In that case, I would suggest looking for any configuration/setting in your test framework to enable it return a non-zero exit code for failed tests.

I hope that helps! Let us know in case you have any questions.

Thank you, @Adam Brunt !

Patrik S