Run and parse MSTest with multiple containers

Alfredo Hidalgo

Hello, I'm trying to achive the following:

Run MSTest with multiple containers and get the test result in Build result. (ex: 11 Tests, 3 failed)

The point is the current MSTest task only accept one container, I want to run all ùy test assemblies in one row to generate only one testresult file.

Also, I dont want to explicit specify all containers because I want to avoid to reconfigure Bamboo task each time a new test assembly is added to solution.

So, my current approach is to execute MSTest from a custom MSBuild task that will look for *.Test.dll assemblies and generate a command line with multiple "/container" parameters. This is running ok but when a test fail, the MSBuild task fail (exit code = 1, the MSTest exit code actually) and Bamboo assume build is failing and dont show test results information (ex 11 Test executed, 3 failed)

So, any one can give a hint on it ?

Scott Guymer

I came up with a solution to this in the end

I created an MSBuild task to get all the names of the tests, run mstest and save the output to a file

Then use the test parser to pick up the file and parse the tests

here is the build XML which i saved as a .proj file in the troot of the repository (but coudl go anywhere althought you might have to change the include mask in the build file)

<Project ToolsVersion="4.0" DefaultTarget="RunTests" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <PropertyGroup>
    <results_file>./TestResults/results.trx</results_file>
  </PropertyGroup>

  <PropertyGroup>
    <MsTestExePath Condition="'$(MsTestExePath)'==''">C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe</MsTestExePath>
  </PropertyGroup>

  <ItemGroup>
    <TestAssemblies Include=".\**\bin\**\*.Tests.dll" />
  </ItemGroup>

  <ItemGroup>
    <ResultsFile Include="$(results_file)" />
  </ItemGroup>

  <Target Name="RunTests">
    <Message Text="Found test assemblies: @(TestAssemblies)" />

    <MakeDir Directories="./TestResults" />
    <CallTarget Targets="RunMsTest" />
  </Target>

  <Target Name="RunMsTest" >
    <Message Text="Running to results file: %(ResultsFile.Identity)" />

    <PropertyGroup>
      <MsTestCommand>"$(MsTestExePath)" @(TestAssemblies ->'/testcontainer:"%(RecursiveDir)%(Filename)%(Extension)" ', ' ') /resultsfile:TestResults\%(ResultsFile.Identity)</MsTestCommand>
    </PropertyGroup>

    <Message Text="Running command $(MsTestCommand)" />

    <Exec Condition=" '@(TestAssemblies)' != ''" Command="$(MsTestCommand)" ContinueOnError="true">
      <Output TaskParameter="ExitCode" ItemName="ErrorCode"/>
    </Exec>
    <Message Text="Tests complete" />
    <Message Text="The exit code is $(ErrorCode)"/>
    <Error Text="Error while executing MSTest" Condition="'$(ErrorCode)' != ''" />
    <OnError ExecuteTargets="MessageErrorHandler"/>       
  </Target>

  <Target Name="MessageErrorHandler">
    <Message Text="An error has occurred while executing MSTest"/>
  </Target>

</Project>

what i then did was create an msbuild task in bamboo and set it to build the proj file. then set the following in the options line

/p:results_file="testresults-${bamboo.buildKey}-${bamboo.buildNumber}.trx"

this tells it which file name to use.

I then created an mstest parser task to pick up the file

testresults-${bamboo.buildKey}-${bamboo.buildNumber}.trx

took a little trial and error but works great. Even spits out the test results into the log..