Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Failing tests not failing Pipeline?

I have recently noticed that if I run tests in my pipeline and some tests are failing, that my pipeline will report the failed tests correctly, but won't stop. It happily continues with defective code, in my case deploying it to Staging.

I honestly would expect a CI/CD Pipeline to fail, rather than to ignore it.

 

What do I do: I run jasmine tests that write to an xml file and i make sure that this file is picked up by pipelines as described here:  https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html

As you can see in the screenshot, it detects my reporting and correctly marks some tests as tailed. But the step itself does not fail.

I cannot find an option to change this behaviour.

Screenshot 2019-10-24 at 10.47.35.png

2 answers

1 accepted

1 vote
Answer accepted

Hi @Bastian Brodbeck, which testing framework do you use? Test reporting is somewhat independent from pipelines being successful or failed. In general, the pipeline will only fail if the exit code of the bash command is non-zero. However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

However, test reporting only shows you the amount of tests that passed or failed and doesn't affect the build status.

I think this is an important information that I could not find anywhere! I would have expected that if any kind of reporter/reporting picked up by Pipelines shows an error it would fail the step.

Since Pipelines clearly detects failed tests in the report I did not expect to have to throw a non-zero-exit code as well. 

I guess I will have to fix that then.

Like # people like this

To me that sounds like a bug and inconsistency across atlassian products. What is the purpose of marking build "successful" if there are failing tests? In bamboo "jUnit report parser" fails the build if there is test failure.

I have to implement my own test result parsing just to fail a build. This is very inconvenient.

I understand that there might be use cases for not failing the build, IMHO most people expect it to fail. I think it should be configurable.

Here is a one liner to fail the build, if someone is looking for a solution.

for file in test-reports/*; do if [ "$(grep -c '<failure' "${file}")" -gt 0 ]; then exit 1; fi; done

 

Like # people like this

I am missing the feature of the bamboo "jUnit report parser" too.

Yes by default bitbucket do not provides anything as such but you can manually create an error if your test case fails similar to this

 

./manage.py test

res=$?

if [ $res -ne 0 ]; then
exit 125
else
echo "Passed"
fi

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket Pipelines

What We Learned When We Researched Open Source Vulnerabilities in 7 Popular Coding Languages

...hey are a part of us, shaping how we interact with the world around us. The same holds true for programming languages when we think about how different kinds of vulnerabilities raise their heads in t...

1,218 views 0 3
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you