Failing tests not failing Pipeline?

[Bastian Brodbeck]

I have recently noticed that if I run tests in my pipeline and some tests are failing, that my pipeline will report the failed tests correctly, but won't stop. It happily continues with defective code, in my case deploying it to Staging.

I honestly would expect a CI/CD Pipeline to fail, rather than to ignore it.

What do I do: I run jasmine tests that write to an xml file and i make sure that this file is picked up by pipelines as described here:  https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html

As you can see in the screenshot, it detects my reporting and correctly marks some tests as tailed. But the step itself does not fail.

I cannot find an option to change this behaviour.

[Alexander Zhukov · Accepted solution]

Hi @Bastian Brodbeck, which testing framework do you use? Test reporting is somewhat independent from pipelines being successful or failed. In general, the pipeline will only fail if the exit code of the bash command is non-zero. However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

[Bastian Brodbeck]

However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

I think this is an important information that I could not find anywhere! I would have expected that if any kind of reporter/reporting picked up by Pipelines shows an error it would fail the step.

Since Pipelines clearly detects failed tests in the report I did not expect to have to throw a non-zero-exit code as well. 

I guess I will have to fix that then.

[Sergey Mokhov]

To me that sounds like a bug and inconsistency across atlassian products. What is the purpose of marking build "successful" if there are failing tests? In bamboo "jUnit report parser" fails the build if there is test failure.

I have to implement my own test result parsing just to fail a build. This is very inconvenient.

I understand that there might be use cases for not failing the build, IMHO most people expect it to fail. I think it should be configurable.

Here is a one liner to fail the build, if someone is looking for a solution.

for file in test-reports/*; do if [ "$(grep -c '<failure' "${file}")" -gt 0 ]; then exit 1; fi; done

[Lukas Burger]

I am missing the feature of the bamboo "jUnit report parser" too.

[Cord Slatton]

@Alexander Zhukov - It would be really great to have an option that could be toggled in the UI that chose whether failing tests caused a build failure, allowing us to stop the pipeline for failed tests really clearly, so the pipeline is more self documenting for behavior. This post has a great way to accomplish what is needed, but it isn't clear then in the UI which of your pipeline steps are supposed to fail tests and fail the build and which are not without looking at at each pipeline file.

[gaurav_pandey]

Yes by default bitbucket do not provides anything as such but you can manually create an error if your test case fails similar to this

./manage.py test

res=$?

if [ $res -ne 0 ]; then
 exit 125
else
 echo "Passed"
fi

[Jason Harrison]

it is unfortunate that:

pytest ; echo $?

outputs 1 when a test fails, and does NOT stop the pipeline

meanwhile, because bitbucket-pipeline scripts are not shell scripts, the following neither outputs 1 nor stops the pipeline:

- pytest
- echo $?

We have in our bash shell scripts:

set -o errexit -o errtrace -o nounset -o pipefail

but this is not sufficient to make a test failing under pytest to fail the pipeline

Other errors fail the pipeline. So either pytest is wrapped or Atlassian is doing something to ignore these errors.