Failing tests not failing Pipeline?

Bastian Brodbeck October 24, 2019

I have recently noticed that if I run tests in my pipeline and some tests are failing, that my pipeline will report the failed tests correctly, but won't stop. It happily continues with defective code, in my case deploying it to Staging.

I honestly would expect a CI/CD Pipeline to fail, rather than to ignore it.

 

What do I do: I run jasmine tests that write to an xml file and i make sure that this file is picked up by pipelines as described here:  https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html

As you can see in the screenshot, it detects my reporting and correctly marks some tests as tailed. But the step itself does not fail.

I cannot find an option to change this behaviour.

Screenshot 2019-10-24 at 10.47.35.png

2 answers

1 accepted

1 vote
Answer accepted
Alexander Zhukov
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 25, 2019

Hi @Bastian Brodbeck, which testing framework do you use? Test reporting is somewhat independent from pipelines being successful or failed. In general, the pipeline will only fail if the exit code of the bash command is non-zero. However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

Bastian Brodbeck October 28, 2019

However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

I think this is an important information that I could not find anywhere! I would have expected that if any kind of reporter/reporting picked up by Pipelines shows an error it would fail the step.

Since Pipelines clearly detects failed tests in the report I did not expect to have to throw a non-zero-exit code as well. 

I guess I will have to fix that then.

Like # people like this
Sergey Mokhov February 3, 2020

To me that sounds like a bug and inconsistency across atlassian products. What is the purpose of marking build "successful" if there are failing tests? In bamboo "jUnit report parser" fails the build if there is test failure.

I have to implement my own test result parsing just to fail a build. This is very inconvenient.

I understand that there might be use cases for not failing the build, IMHO most people expect it to fail. I think it should be configurable.

Here is a one liner to fail the build, if someone is looking for a solution.

for file in test-reports/*; do if [ "$(grep -c '<failure' "${file}")" -gt 0 ]; then exit 1; fi; done

 

Like # people like this
Lukas Burger July 31, 2020

I am missing the feature of the bamboo "jUnit report parser" too.

Like Daniel Baber likes this
Cord Slatton June 23, 2021

@Alexander Zhukov - It would be really great to have an option that could be toggled in the UI that chose whether failing tests caused a build failure, allowing us to stop the pipeline for failed tests really clearly, so the pipeline is more self documenting for behavior. This post has a great way to accomplish what is needed, but it isn't clear then in the UI which of your pipeline steps are supposed to fail tests and fail the build and which are not without looking at at each pipeline file.

0 votes
gaurav_pandey November 3, 2020

Yes by default bitbucket do not provides anything as such but you can manually create an error if your test case fails similar to this

 

./manage.py test

res=$?

if [ $res -ne 0 ]; then
exit 125
else
echo "Passed"
fi
Jason Harrison September 18, 2023

it is unfortunate that:

pytest ; echo $?

outputs 1 when a test fails, and does NOT stop the pipeline

meanwhile, because bitbucket-pipeline scripts are not shell scripts, the following neither outputs 1 nor stops the pipeline:

- pytest
- echo $?

We have in our bash shell scripts:

set -o errexit -o errtrace -o nounset -o pipefail
but this is not sufficient to make a test failing under pytest to fail the pipeline
Other errors fail the pipeline. So either pytest is wrapped or Atlassian is doing something to ignore these errors.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events