On Bamboo I have MSBuild job for building and running tests.
It works fine - but I want to use test quarantining option so when I have some test moved to quarantine (and this moved tests are only what fail) job will be marked as successful
I moved one failing test to quarantine but job is marked as failed
Configuration:
Job in Tasks I have Command named Test runner:
Executable = VSTest.Console
Argument = "WebServiceTestClient.dll" /Logger:trx
And after running job with failing test quarantined I get log
01-Apr-201410:59:44Total tests:62.Passed:61.Failed:1.Skipped:0.
01-Apr-201410:59:44TestRunFailed.01-Apr-201410:59:44Test execution time:10.1296Minutes
01-Apr-201410:59:44Failing task since return code of [C:\Program Files(x86)\Microsoft VisualStudio11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\VSTest.Console.exe "WebServiceTestClient.dll"/Logger:trx] was 1while expected 0
01-Apr-201410:59:44Finished task 'Test runner'
01-Apr-201410:59:44Starting task 'Parse test logs' of type 'com.atlassian.bamboo.plugin.dotnet:mstest'
01-Apr-201410:59:44Parsing test results...
01-Apr-201410:59:45Failing task since 1 failing test cases were found
Job is marked failed but then:
01-Apr-201410:59:45ChangingTaskResult to SUCCESS as all failed tests were quarantined.01-Apr-201410:59:45Finished task 'Parse test logs'01-Apr-201410:59:45Running post build plugin 'NCover Results Collector'01-Apr-201410:59:45Running post build plugin 'Clover Results Collector'01-Apr-201410:59:45Running post build plugin 'Artifact Copier'01-Apr-201410:59:45Finalising the build...
Unfortunately the output is:
This question was asked here http://stackoverflow.com/questions/22803557/prevent-bamboo-from-failing-job-when-failing-tests-are-moved-to-quarantinebut hopefully somebody here give me right answer. |
A fixed plugin is available here:
https://maven.atlassian.com/public/com/atlassian/bamboo/plugins/dotnet/atlassian-bamboo-plugin-dotnet/5.5.2/atlassian-bamboo-plugin-dotnet-5.5.2.jar
You can install it on older Bamboo versions, but first, remove the old plugin from bundled-plugins.zip
There is newer version fixing additional problem when validadtion error occur. Not related to this but you may want to install the newest one.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
We had this problem 2 weeks ago, I just had it run through a script then used the MSTest Parser instead. Yes, admittedingly it is a hack for functionality that should be builtin but it's not a show stopper anymore.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Would you mind providing more details about this script and configuration? We are admittedly a *nix shop but maintaining a Windows app during a migration process, so we defintely aren't windows scripting experts. How do you force the script to return 0?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
The script always return 0 by default, you have to do extra work to make it fail a build. It just runs MSTest with testContainer, resultsFile, & category command line option. Then I use the MSTest Parser task to pickup the .trx file I just generated.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I'm going to need a little more detail. I tried doing this and it is definitely still returning an error code and it doesn't even make it to the MSTest Parser task. Here is the inline script I was using...
echo off echo Running Tests writing results to testresults-%BAMBOO_BUILD_NUMBER%.trx "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:./Source/UnitTest.DAL/bin/Debug/DSC.Dal.UnitTest.dll /resultsfile:testresults-%BAMBOO_BUILD_NUMBER%.trx
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
It has to do with the fact that the MSTest Runner script is still returning non-zero result because there is a failing test. Bamboo sees this non-zero result from the script and assumes there is an issue. I'm surprised this doesn't work since Atlassian provides the MSTest Runner script. In my opinion this is a bug so I'm creating a support request with Atlassian.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.