Prevent Bamboo from failing job when failing tests are moved to quarantine

On Bamboo I have MSBuild job for building and running tests.

It works fine - but I want to use test quarantining option so when I have some test moved to quarantine (and this moved tests are only what fail) job will be marked as successful

I moved one failing test to quarantine but job is marked as failed

Configuration:

Job in Tasks I have Command named Test runner:

Executable = VSTest.Console

Argument = "WebServiceTestClient.dll" /Logger:trx

And after running job with failing test quarantined I get log

01-Apr-201410:59:44Total tests:62.Passed:61.Failed:1.Skipped:0.
01-Apr-201410:59:44TestRunFailed.01-Apr-201410:59:44Test execution time:10.1296Minutes
01-Apr-201410:59:44Failing task since return code of [C:\Program Files(x86)\Microsoft VisualStudio11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\VSTest.Console.exe "WebServiceTestClient.dll"/Logger:trx] was 1while expected 0
01-Apr-201410:59:44Finished task 'Test runner'
01-Apr-201410:59:44Starting task 'Parse test logs' of type 'com.atlassian.bamboo.plugin.dotnet:mstest'
01-Apr-201410:59:44Parsing test results...
01-Apr-201410:59:45Failing task since 1 failing test cases were found

Job is marked failed but then:

01-Apr-201410:59:45ChangingTaskResult to SUCCESS as all failed tests were quarantined.01-Apr-201410:59:45Finished task 'Parse test logs'01-Apr-201410:59:45Running post build plugin 'NCover Results Collector'01-Apr-201410:59:45Running post build plugin 'Clover Results Collector'01-Apr-201410:59:45Running post build plugin 'Artifact Copier'01-Apr-201410:59:45Finalising the build...

Unfortunately the output is:

enter image description here

This question was asked here http://stackoverflow.com/questions/22803557/prevent-bamboo-from-failing-job-when-failing-tests-are-moved-to-quarantinebut hopefully somebody here give me right answer.

3 answers

We had this problem 2 weeks ago, I just had it run through a script then used the MSTest Parser instead. Yes, admittedingly it is a hack for functionality that should be builtin but it's not a show stopper anymore.

Would you mind providing more details about this script and configuration? We are admittedly a *nix shop but maintaining a Windows app during a migration process, so we defintely aren't windows scripting experts. How do you force the script to return 0?

The script always return 0 by default, you have to do extra work to make it fail a build. It just runs MSTest with testContainer, resultsFile, & category command line option. Then I use the MSTest Parser task to pickup the .trx file I just generated.

I'm going to need a little more detail. I tried doing this and it is definitely still returning an error code and it doesn't even make it to the MSTest Parser task. Here is the inline script I was using...

echo off
echo Running Tests writing results to testresults-%BAMBOO_BUILD_NUMBER%.trx
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:./Source/UnitTest.DAL/bin/Debug/DSC.Dal.UnitTest.dll /resultsfile:testresults-%BAMBOO_BUILD_NUMBER%.trx

1 vote

A fixed plugin is available here:

https://maven.atlassian.com/public/com/atlassian/bamboo/plugins/dotnet/atlassian-bamboo-plugin-dotnet/5.5.2/atlassian-bamboo-plugin-dotnet-5.5.2.jar

You can install it on older Bamboo versions, but first, remove the old plugin from bundled-plugins.zip

There is newer version fixing additional problem when validadtion error occur. Not related to this but you may want to install the newest one.

https://maven.atlassian.com/public/com/atlassian/bamboo/plugins/dotnet/atlassian-bamboo-plugin-dotnet/5.5.3/atlassian-bamboo-plugin-dotnet-5.5.3.jar

It has to do with the fact that the MSTest Runner script is still returning non-zero result because there is a failing test. Bamboo sees this non-zero result from the script and assumes there is an issue. I'm surprised this doesn't work since Atlassian provides the MSTest Runner script. In my opinion this is a bug so I'm creating a support request with Atlassian.

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published May 18, 2017 in Bamboo

FAQ: How to Upgrade Bamboo Server

Bamboo 5.9 will no longer be supported after June 12, 2017. What does this mean? As part of our End of Life policy, Atlassian supports major versions for two years after the first major iteratio...

1,568 views 0 6
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you