Prevent Bamboo from failing job when failing tests are moved to quarantine

pbaranski April 10, 2014

On Bamboo I have MSBuild job for building and running tests.

It works fine - but I want to use test quarantining option so when I have some test moved to quarantine (and this moved tests are only what fail) job will be marked as successful

I moved one failing test to quarantine but job is marked as failed

Configuration:

Job in Tasks I have Command named Test runner:

Executable = VSTest.Console

Argument = "WebServiceTestClient.dll" /Logger:trx

And after running job with failing test quarantined I get log

01-Apr-201410:59:44Total tests:62.Passed:61.Failed:1.Skipped:0.
01-Apr-201410:59:44TestRunFailed.01-Apr-201410:59:44Test execution time:10.1296Minutes
01-Apr-201410:59:44Failing task since return code of [C:\Program Files(x86)\Microsoft VisualStudio11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\VSTest.Console.exe "WebServiceTestClient.dll"/Logger:trx] was 1while expected 0
01-Apr-201410:59:44Finished task 'Test runner'
01-Apr-201410:59:44Starting task 'Parse test logs' of type 'com.atlassian.bamboo.plugin.dotnet:mstest'
01-Apr-201410:59:44Parsing test results...
01-Apr-201410:59:45Failing task since 1 failing test cases were found

Job is marked failed but then:

01-Apr-201410:59:45ChangingTaskResult to SUCCESS as all failed tests were quarantined.01-Apr-201410:59:45Finished task 'Parse test logs'01-Apr-201410:59:45Running post build plugin 'NCover Results Collector'01-Apr-201410:59:45Running post build plugin 'Clover Results Collector'01-Apr-201410:59:45Running post build plugin 'Artifact Copier'01-Apr-201410:59:45Finalising the build...

Unfortunately the output is:

enter image description here

This question was asked here http://stackoverflow.com/questions/22803557/prevent-bamboo-from-failing-job-when-failing-tests-are-moved-to-quarantinebut hopefully somebody here give me right answer.

3 answers

1 vote
Przemek Bruski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 5, 2014

A fixed plugin is available here:

https://maven.atlassian.com/public/com/atlassian/bamboo/plugins/dotnet/atlassian-bamboo-plugin-dotnet/5.5.2/atlassian-bamboo-plugin-dotnet-5.5.2.jar

You can install it on older Bamboo versions, but first, remove the old plugin from bundled-plugins.zip

Pawel Skierczynski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 12, 2014

There is newer version fixing additional problem when validadtion error occur. Not related to this but you may want to install the newest one.

https://maven.atlassian.com/public/com/atlassian/bamboo/plugins/dotnet/atlassian-bamboo-plugin-dotnet/5.5.3/atlassian-bamboo-plugin-dotnet-5.5.3.jar

1 vote
Jason Monsorno
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
May 29, 2014

We had this problem 2 weeks ago, I just had it run through a script then used the MSTest Parser instead. Yes, admittedingly it is a hack for functionality that should be builtin but it's not a show stopper anymore.

Matt Block May 29, 2014

Would you mind providing more details about this script and configuration? We are admittedly a *nix shop but maintaining a Windows app during a migration process, so we defintely aren't windows scripting experts. How do you force the script to return 0?

Jason Monsorno
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
May 29, 2014

The script always return 0 by default, you have to do extra work to make it fail a build. It just runs MSTest with testContainer, resultsFile, & category command line option. Then I use the MSTest Parser task to pickup the .trx file I just generated.

Matthew Block May 30, 2014

I'm going to need a little more detail. I tried doing this and it is definitely still returning an error code and it doesn't even make it to the MSTest Parser task. Here is the inline script I was using...

echo off
echo Running Tests writing results to testresults-%BAMBOO_BUILD_NUMBER%.trx
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:./Source/UnitTest.DAL/bin/Debug/DSC.Dal.UnitTest.dll /resultsfile:testresults-%BAMBOO_BUILD_NUMBER%.trx

0 votes
Matt Block May 29, 2014

It has to do with the fact that the MSTest Runner script is still returning non-zero result because there is a failing test. Bamboo sees this non-zero result from the script and assumes there is an issue. I'm surprised this doesn't work since Atlassian provides the MSTest Runner script. In my opinion this is a bug so I'm creating a support request with Atlassian.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events