Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Failing tests not failing Pipeline?

I have recently noticed that if I run tests in my pipeline and some tests are failing, that my pipeline will report the failed tests correctly, but won't stop. It happily continues with defective code, in my case deploying it to Staging.

I honestly would expect a CI/CD Pipeline to fail, rather than to ignore it.

 

What do I do: I run jasmine tests that write to an xml file and i make sure that this file is picked up by pipelines as described here:  https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html

As you can see in the screenshot, it detects my reporting and correctly marks some tests as tailed. But the step itself does not fail.

I cannot find an option to change this behaviour.

Screenshot 2019-10-24 at 10.47.35.png

2 answers

1 accepted

1 vote
Answer accepted

Hi @Bastian Brodbeck, which testing framework do you use? Test reporting is somewhat independent from pipelines being successful or failed. In general, the pipeline will only fail if the exit code of the bash command is non-zero. However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

I think this is an important information that I could not find anywhere! I would have expected that if any kind of reporter/reporting picked up by Pipelines shows an error it would fail the step.

Since Pipelines clearly detects failed tests in the report I did not expect to have to throw a non-zero-exit code as well. 

I guess I will have to fix that then.

Like # people like this

To me that sounds like a bug and inconsistency across atlassian products. What is the purpose of marking build "successful" if there are failing tests? In bamboo "jUnit report parser" fails the build if there is test failure.

I have to implement my own test result parsing just to fail a build. This is very inconvenient.

I understand that there might be use cases for not failing the build, IMHO most people expect it to fail. I think it should be configurable.

Here is a one liner to fail the build, if someone is looking for a solution.

for file in test-reports/*; do if [ "$(grep -c '<failure' "${file}")" -gt 0 ]; then exit 1; fi; done

 

Like # people like this

I am missing the feature of the bamboo "jUnit report parser" too.

@Alexander Zhukov - It would be really great to have an option that could be toggled in the UI that chose whether failing tests caused a build failure, allowing us to stop the pipeline for failed tests really clearly, so the pipeline is more self documenting for behavior. This post has a great way to accomplish what is needed, but it isn't clear then in the UI which of your pipeline steps are supposed to fail tests and fail the build and which are not without looking at at each pipeline file.

Yes by default bitbucket do not provides anything as such but you can manually create an error if your test case fails similar to this

 

./manage.py test

res=$?

if [ $res -ne 0 ]; then
exit 125
else
echo "Passed"
fi

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket Pipelines

Bitbucket Pipelines Runners is now in open beta

We are excited to announce the open beta program for self-hosted runners. Bitbucket Pipelines Runners is available to everyone. Please try it and let us know your feedback. If you have any issue...

3,805 views 64 20
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you