Clover ignore -Dmaven.test.failure.ignore=false

[Baron Roberts]

As part of our nightly Jenkins build we are running Clover via the Maven plugin. We have experienced a unit test succeeding when run outside Clover but failing when run instrumented by Clover. The Jenkins job has -Dmaven.test.failure.ignore=false specified as part of the MAVEN_OPTS. This setting fails builds when the unit tests fail. However, we are seeing that this property setting is not being honored when Clover runs the unit test. In that case, the build does not fail, it is just marked unstable. Are MAVEN_OPTS not being honored when clover runs a build/test? Is there some way to pass MAVEN_OPTS to Clover?

[Marek Parfianowicz · Accepted solution]

The CLOV-1109 was implemented in 3.1.10 - you can define setTestFailureIgnore property for Clover. See Changes+in+3.1.10+for+Maven+2+and+3 for more details.

[Marek Parfianowicz]

Please compare following two examples:

1) clover2:setup + clover2:clover - it will fail a build as expected

<plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>false</testFailureIgnore>
                </configuration>
            </plugin>
            <plugin>
                <groupId>com.atlassian.maven.plugins</groupId>
                <artifactId>maven-clover2-plugin</artifactId>
                <version>3.1.6</version>
                <configuration>
                    <licenseLocation>${clover.licenseLocation}</licenseLocation>
                    <generateHtml>true</generateHtml>
                </configuration>
                <executions>
                    <execution>
                        <id>init</id>
                        <phase>initialize</phase>
                        <goals>
                            <goal>setup</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>report</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>clover</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>

2) clover2:instrument + clover2:clover - it will not fail build in a parallel lifecycle, but fail in main cycle

<plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>false</testFailureIgnore>
                </configuration>
            </plugin>
            <plugin>
                <groupId>com.atlassian.maven.plugins</groupId>
                <artifactId>maven-clover2-plugin</artifactId>
                <version>3.1.6</version>
                <configuration>
                    <licenseLocation>${clover.licenseLocation}</licenseLocation>
                    <generateHtml>true</generateHtml>
                </configuration>
                <executions>
                    <execution>
                        <id>report</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>instrument</goal>
                            <goal>clover</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>

Execute using: mvn clean install

[Baron Roberts]

Thanks for the suggestions Marek. Currently, we only want to run Clover during site generation. So we run the "instrument-test" goal during the "pre-site" phase. We need to do "instrument-test" to avoid our Findbugs static analyzer failing the build analyzing the Clover instrumented source. How would you adapt the use of "setup" and "clover" to our situation?

As far as lifecycle.xml goes, it seems like a bad idea to alter the configuration of the Surefire plugin from the Clover2 plugin. Are you folks planning to change this "HACK" to a configuration parameter rather than a hard coded override?

[Marek Parfianowicz]

Have you considered using profiles and running build twice: one with Clover and one with Findbugs?

[Baron Roberts]

We do use profiles and can consider separate builds but that is a lot of overhead for us for working around a bug in the Clover plugin. I would be more inclined to fork the plugin, apply a patch to fix the bug and use that in our builds.

Can I open a JIRA issue against the Clover2 plugin to address this bug and if so, could you provide the link?

[Baron Roberts]

I have created CLOV-1109 for this issue.

[Marek Parfianowicz]

Hi Baron,

Clover does not execute tests on his own - it just instruments sources, which are later being executed by test plugin. So typically, a standard maven-surefire plugin is running unit tests. And the surefire plugin reads MAVEN_OPTS options of course.

My suggestion is to have a deeper look at the test case - why it passes when it's not instrumented and why it fails when instrumented with Clover. There might be subtle differences related with exception handling inside a test case. Does your test case fail because of failed assertion or fails because of JUnit framework error (e.g. unexpected exception thrown)? Do you use @Test(expected=SomeException.class) or @ExpectedException(...) tags for example?

Could you also explain what do you mean by "the build does not fail, it is just marked unstable"?

Regards
Marek

[Baron Roberts]

Hi Marek,

Thanks for your reply. The problem is not that the test cases fail when run by Clover. Rather, it is that the surefire plugin does not fail the build even though it has been configured to do so. When surefire runs the unit tests outside the Clvoer lifecycle, it will fail the build if the unit tests fail. However, when surefire runs the instrumented tests within the Clover lifecycle, it does not fail the build when a unit test fails. It is as if the surefire plugin configuration in the Clover lifecycle is not picking up the configuration.

Specifically, in Jenkins we have specified MAVEN_OPTS with -Dmaven.test.failure.ignore=false. This results in the build failing if there are unit test failures. This works as expected if there are unit test failures during the normal, non-Clover lifecycle run. If the tests fail in the Clover lifecycle portion of the build, the build does not fail (i.e. yellow status instead of red). You might ask why the tests would not fail during the normal build but fail after Clover instrumentation. We find that running under Clover can inadvertently reorder tests and any order depedency bugs in the tests are surfaced. So the failure of the tests is fine, the fact that it does not fail the build is not.

[Baron Roberts]

I think I have found the problem. Looking at your source code for the Clover Maven plugin, I find the folowing in the file src/main/resources/META-INF/maven/lifecycle.xml:

<!-- This below is a HACK. It tries to prevent the build from stopping in
           case of build failures as we could still like to generate Clover
           reports even if it happens. The Clover plugin cannot know about all
           plugins that can execute and fail the build and moreover which can be
           bound to any phase by the user. A better solution would be for Maven
           to have a flag that says "continue the build even in case of
           failures". -->
      <phase>
        <id>test</id>
        <configuration>
          <testFailureIgnore>true</testFailureIgnore>
        </configuration>
      </phase>
      <phase>
        <id>integration-test</id>
        <configuration>
          <testFailureIgnore>true</testFailureIgnore>
        </configuration>
      </phase>

This agrees with my findings that the tests run outside of Clover's lifecycle fail the build but tests run within Clover's lifecycle do not. Effectively, your plugin is overriding the surefire plugin's configuration. Any suggestions on how to override your override?

[Marek Parfianowicz]

Hi Baron,

Thank you for providing more details. Please note that we have two Clover goals for source code instrumentation: clover2:setup and clover2:instrument.

The clover2:instrument forks a parallel build lifecycle as you can find in CloverInstrumentMojo:

/**
 * [...]
 * @goal instrument
 * @execute phase="install" lifecycle="clover"
 */
public class CloverInstrumentMojo extends AbstractCloverMojo

The lifecycle="clover" causes to use the definition from lifecycle.xml and thus forces to use <testFailureIgnore>true</testFailureIgnore>. Unfortunately, it cannot be overridden by other maven property.

However, as I mentioned we've got a second goal named clover2:setup. It does not fork a parallel build lifecycle and thus it will honour settings used in your pom.xml or mvn command line.