nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Failing tests not failing Pipeline?

Bastian Brodbeck

I have recently noticed that if I run tests in my pipeline and some tests are failing, that my pipeline will report the failed tests correctly, but won't stop. It happily continues with defective code, in my case deploying it to Staging.

I honestly would expect a CI/CD Pipeline to fail, rather than to ignore it.

What do I do: I run jasmine tests that write to an xml file and i make sure that this file is picked up by pipelines as described here: https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html

As you can see in the screenshot, it detects my reporting and correctly marks some tests as tailed. But the step itself does not fail.

I cannot find an option to change this behaviour.

Screenshot 2019-10-24 at 10.47.35.png

Accepted answers

Alexander Zhukov_1bqyxb

Hi @Bastian Brodbeck, which testing framework do you use? Test reporting is somewhat independent from pipelines being successful or failed. In general, the pipeline will only fail if the exit code of the bash command is non-zero. However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

All comments

Alexander Zhukov_1bqyxb

Bastian Brodbeck

However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

I think this is an important information that I could not find anywhere! I would have expected that if any kind of reporter/reporting picked up by Pipelines shows an error it would fail the step.

Since Pipelines clearly detects failed tests in the report I did not expect to have to throw a non-zero-exit code as well.

I guess I will have to fix that then.

Sergey Mokhov

To me that sounds like a bug and inconsistency across atlassian products. What is the purpose of marking build "successful" if there are failing tests? In bamboo "jUnit report parser" fails the build if there is test failure.

I have to implement my own test result parsing just to fail a build. This is very inconvenient.

I understand that there might be use cases for not failing the build, IMHO most people expect it to fail. I think it should be configurable.

Here is a one liner to fail the build, if someone is looking for a solution.

for file in test-reports/*; do if [ "$(grep -c '<failure' "${file}")" -gt 0 ]; then exit 1; fi; done

Lukas Burger

I am missing the feature of the bamboo "jUnit report parser" too.

gaurav_pandey_zwr57g

Yes by default bitbucket do not provides anything as such but you can manually create an error if your test case fails similar to this

./manage.py test


res=$?

if [ $res -ne 0 ]; then
 exit 125
else
 echo "Passed"
fi

Jason Harrison_dno8te

it is unfortunate that:

pytest ; echo $?

outputs 1 when a test fails, and does NOT stop the pipeline

meanwhile, because bitbucket-pipeline scripts are not shell scripts, the following neither outputs 1 nor stops the pipeline:

- pytest
- echo $?

We have in our bash shell scripts:

set -o errexit -o errtrace -o nounset -o pipefail

but this is not sufficient to make a test failing under pytest to fail the pipeline

Other errors fail the pipeline. So either pytest is wrapped or Atlassian is doing something to ignore these errors.