Run and parse MSTest with multiple containers

Hello, I'm trying to achive the following:

Run MSTest with multiple containers and get the test result in Build result. (ex: 11 Tests, 3 failed)

The point is the current MSTest task only accept one container, I want to run all ùy test assemblies in one row to generate only one testresult file.

Also, I dont want to explicit specify all containers because I want to avoid to reconfigure Bamboo task each time a new test assembly is added to solution.

So, my current approach is to execute MSTest from a custom MSBuild task that will look for *.Test.dll assemblies and generate a command line with multiple "/container" parameters. This is running ok but when a test fail, the MSBuild task fail (exit code = 1, the MSTest exit code actually) and Bamboo assume build is failing and dont show test results information (ex 11 Test executed, 3 failed)

So, any one can give a hint on it ?

4 answers

1 accepted

This widget could not be displayed.

I came up with a solution to this in the end

I created an MSBuild task to get all the names of the tests, run mstest and save the output to a file

Then use the test parser to pick up the file and parse the tests

here is the build XML which i saved as a .proj file in the troot of the repository (but coudl go anywhere althought you might have to change the include mask in the build file)

<Project ToolsVersion="4.0" DefaultTarget="RunTests" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <PropertyGroup>
    <results_file>./TestResults/results.trx</results_file>
  </PropertyGroup>

  <PropertyGroup>
    <MsTestExePath Condition="'$(MsTestExePath)'==''">C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe</MsTestExePath>
  </PropertyGroup>

  <ItemGroup>
    <TestAssemblies Include=".\**\bin\**\*.Tests.dll" />
  </ItemGroup>

  <ItemGroup>
    <ResultsFile Include="$(results_file)" />
  </ItemGroup>

  <Target Name="RunTests">
    <Message Text="Found test assemblies: @(TestAssemblies)" />

    <MakeDir Directories="./TestResults" />
    <CallTarget Targets="RunMsTest" />
  </Target>

  <Target Name="RunMsTest" >
    <Message Text="Running to results file: %(ResultsFile.Identity)" />

    <PropertyGroup>
      <MsTestCommand>"$(MsTestExePath)" @(TestAssemblies ->'/testcontainer:"%(RecursiveDir)%(Filename)%(Extension)" ', ' ') /resultsfile:TestResults\%(ResultsFile.Identity)</MsTestCommand>
    </PropertyGroup>

    <Message Text="Running command $(MsTestCommand)" />

    <Exec Condition=" '@(TestAssemblies)' != ''" Command="$(MsTestCommand)" ContinueOnError="true">
      <Output TaskParameter="ExitCode" ItemName="ErrorCode"/>
    </Exec>
    <Message Text="Tests complete" />
    <Message Text="The exit code is $(ErrorCode)"/>
    <Error Text="Error while executing MSTest" Condition="'$(ErrorCode)' != ''" />
    <OnError ExecuteTargets="MessageErrorHandler"/>       
  </Target>

  <Target Name="MessageErrorHandler">
    <Message Text="An error has occurred while executing MSTest"/>
  </Target>

</Project>

what i then did was create an msbuild task in bamboo and set it to build the proj file. then set the following in the options line

/p:results_file="testresults-${bamboo.buildKey}-${bamboo.buildNumber}.trx"

this tells it which file name to use.

I then created an mstest parser task to pick up the file

testresults-${bamboo.buildKey}-${bamboo.buildNumber}.trx

took a little trial and error but works great. Even spits out the test results into the log..

I tried this and it runs all tests successfully however Ms test Parser task is not picking up the file. 

as you mentioned i did little trial and error too however none of my tricks worked. Can you elaborate on what did you do to make it work finally. 

This widget could not be displayed.

Im trying to overcome the same issue. Did you get anywhere?

The only thing i can think of is to runa script to identify all *.test.dll and put them into a bamboo run config variable separated by /testcontainer

Then run a custom task to start mstest with the previous string and output to a path using variables

Then run the test parser...

This widget could not be displayed.

Hello, almost the same solution I applied in my side. Custom MSBuild task, like Scott describe.

Its working ok for me.

This widget could not be displayed.

I currently have the problem that MSTest parser does not seem to work correctly. Of an expected number of around 1300 tests only 650 tests are shown in the bamboo status page of the build. The approach for generating the TRX file is similar to the one mentioned above.

The genenerated TRX file contains all expected tests as well as the correct results, hence the test runner does his jobs. I've looked into the implementation of MSTestXmlTestResultsParser, which should be responsible for parsing trx files. MSTestXmlTestResultsParser does not perform any filtering, hence the problem must be somewhere else.

First I suspected that duplicate testnames are filtered, but that does not seem to be the case. The trx contains indeed duplicate test names, but after filtering duplicates there are only 617 left and not 650. Does anybody have an idea?

Suggest an answer

Log in or Sign up to answer
Atlassian Summit 2018

Meet the community IRL

Atlassian Summit is an excellent opportunity for in-person support, training, and networking.

Learn more
Community showcase
Posted yesterday in Statuspage

What are your best incident management tips and stories? #HugOps

 👋Community members! Downtime happens. And great incident response takes a village. Teams like Support, Dev, SRE, Ops, IT, and Marketing have to come together to resolve the problem while keep...

60 views 2 3
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you