Bamboo build succeeds even if test fails

Question

Hello,

We have a build plan for a .NET 6 solution.

Our Bamboo version is 8.0.0 server edition.

This is our Tasks list:

Script task basically does "dotnet test" of all test projects and outputs results with the --logger parameter to a .trx file.

Problem is, even tho a test fails, this step just continues as successful. Additionally, the MSTest Parser final task parses tests as successful.

Why and how do I fix this?

Jonas Gucu_ta7e7r · Answer

Okay, so I redid our tasks list (not sure if first one is really needed):

Problems and changes:

1. Before we built 2 UI projects separately, that way if a test project had a compilation error, Bamboo could not see it. I swapped building two separate UI projects for the whole solution, now all compilation errors will be caught.

2. "dotnet-test" should fail the build if a test project has a compilation error, but it does not fail. I think this is a bug.

3. Swapped the order: before we used to test first in order not to waste time building, but now solution will be built first and tests ran later.

4. Artifact copy fails if "Clean working directory after each build" flag is checked in plan configuration.

So it seems that Bamboo copies artifacts AFTER cleaning working directory, which doesn't make sense? Should it not clean AFTER everything is done? Because in our case the build directory gets cleaned, artifacts disappear and artifact copy fails.

Jonas Gucu_ta7e7r · Answer

Okay, I think this error maybe not be what I think it is?

I left just the "dotnet test" step.

When "dotnet test" runs the project that has a COMPILATION error (see 2nd image above), it does not even run tests for the project, so no test result is given?

If a project has no compilation error, the test runs just fine and this is output in the log:

The project with the compilation error does not have this output.

Even with the compilation error in "dotnet test", the bamboo build successfully passes.

Charlie Misonne · Answer

A Bamboo task will only fail if the exit code of the last command is 0. Microsoft's documentation says: "If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1"

In you scenario it looks like the "test run" Bamboo task does return a 0 as exit code. Are you performing anything else after the dotnet test command that might return 0?