Clover ignore -Dmaven.test.failure.ignore=false

As part of our nightly Jenkins build we are running Clover via the Maven plugin. We have experienced a unit test succeeding when run outside Clover but failing when run instrumented by Clover. The Jenkins job has -Dmaven.test.failure.ignore=false specified as part of the MAVEN_OPTS. This setting fails builds when the unit tests fail. However, we are seeing that this property setting is not being honored when Clover runs the unit test. In that case, the build does not fail, it is just marked unstable. Are MAVEN_OPTS not being honored when clover runs a build/test? Is there some way to pass MAVEN_OPTS to Clover?

3 answers

1 accepted

This widget could not be displayed.

The CLOV-1109 was implemented in 3.1.10 - you can define setTestFailureIgnore property for Clover. See Changes+in+3.1.10+for+Maven+2+and+3 for more details.

This widget could not be displayed.

Hi Baron,

Clover does not execute tests on his own - it just instruments sources, which are later being executed by test plugin. So typically, a standard maven-surefire plugin is running unit tests. And the surefire plugin reads MAVEN_OPTS options of course.

My suggestion is to have a deeper look at the test case - why it passes when it's not instrumented and why it fails when instrumented with Clover. There might be subtle differences related with exception handling inside a test case. Does your test case fail because of failed assertion or fails because of JUnit framework error (e.g. unexpected exception thrown)? Do you use @Test(expected=SomeException.class) or @ExpectedException(...) tags for example?

Could you also explain what do you mean by "the build does not fail, it is just marked unstable"?

Regards
Marek

Hi Marek,

Thanks for your reply. The problem is not that the test cases fail when run by Clover. Rather, it is that the surefire plugin does not fail the build even though it has been configured to do so. When surefire runs the unit tests outside the Clvoer lifecycle, it will fail the build if the unit tests fail. However, when surefire runs the instrumented tests within the Clover lifecycle, it does not fail the build when a unit test fails. It is as if the surefire plugin configuration in the Clover lifecycle is not picking up the configuration.

Specifically, in Jenkins we have specified MAVEN_OPTS with -Dmaven.test.failure.ignore=false. This results in the build failing if there are unit test failures. This works as expected if there are unit test failures during the normal, non-Clover lifecycle run. If the tests fail in the Clover lifecycle portion of the build, the build does not fail (i.e. yellow status instead of red). You might ask why the tests would not fail during the normal build but fail after Clover instrumentation. We find that running under Clover can inadvertently reorder tests and any order depedency bugs in the tests are surfaced. So the failure of the tests is fine, the fact that it does not fail the build is not.

I think I have found the problem. Looking at your source code for the Clover Maven plugin, I find the folowing in the file src/main/resources/META-INF/maven/lifecycle.xml:

<!-- This below is a HACK. It tries to prevent the build from stopping in
           case of build failures as we could still like to generate Clover
           reports even if it happens. The Clover plugin cannot know about all
           plugins that can execute and fail the build and moreover which can be
           bound to any phase by the user. A better solution would be for Maven
           to have a flag that says "continue the build even in case of
           failures". -->
      <phase>
        <id>test</id>
        <configuration>
          <testFailureIgnore>true</testFailureIgnore>
        </configuration>
      </phase>
      <phase>
        <id>integration-test</id>
        <configuration>
          <testFailureIgnore>true</testFailureIgnore>
        </configuration>
      </phase>

This agrees with my findings that the tests run outside of Clover's lifecycle fail the build but tests run within Clover's lifecycle do not. Effectively, your plugin is overriding the surefire plugin's configuration. Any suggestions on how to override your override?

Hi Baron,

Thank you for providing more details. Please note that we have two Clover goals for source code instrumentation: clover2:setup and clover2:instrument.

The clover2:instrument forks a parallel build lifecycle as you can find in CloverInstrumentMojo:

/**
 * [...]
 * @goal instrument
 * @execute phase="install" lifecycle="clover"
 */
public class CloverInstrumentMojo extends AbstractCloverMojo

The lifecycle="clover" causes to use the definition from lifecycle.xml and thus forces to use <testFailureIgnore>true</testFailureIgnore>. Unfortunately, it cannot be overridden by other maven property.

However, as I mentioned we've got a second goal named clover2:setup. It does not fork a parallel build lifecycle and thus it will honour settings used in your pom.xml or mvn command line.

This widget could not be displayed.

Please compare following two examples:

1) clover2:setup + clover2:clover - it will fail a build as expected

&lt;plugins&gt;
            &lt;plugin&gt;
                &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                &lt;artifactId&gt;maven-surefire-plugin&lt;/artifactId&gt;
                &lt;configuration&gt;
                    &lt;testFailureIgnore&gt;false&lt;/testFailureIgnore&gt;
                &lt;/configuration&gt;
            &lt;/plugin&gt;
            &lt;plugin&gt;
                &lt;groupId&gt;com.atlassian.maven.plugins&lt;/groupId&gt;
                &lt;artifactId&gt;maven-clover2-plugin&lt;/artifactId&gt;
                &lt;version&gt;3.1.6&lt;/version&gt;
                &lt;configuration&gt;
                    &lt;licenseLocation&gt;${clover.licenseLocation}&lt;/licenseLocation&gt;
                    &lt;generateHtml&gt;true&lt;/generateHtml&gt;
                &lt;/configuration&gt;
                &lt;executions&gt;
                    &lt;execution&gt;
                        &lt;id&gt;init&lt;/id&gt;
                        &lt;phase&gt;initialize&lt;/phase&gt;
                        &lt;goals&gt;
                            &lt;goal&gt;setup&lt;/goal&gt;
                        &lt;/goals&gt;
                    &lt;/execution&gt;
                    &lt;execution&gt;
                        &lt;id&gt;report&lt;/id&gt;
                        &lt;phase&gt;validate&lt;/phase&gt;
                        &lt;goals&gt;
                            &lt;goal&gt;clover&lt;/goal&gt;
                        &lt;/goals&gt;
                    &lt;/execution&gt;
                &lt;/executions&gt;
            &lt;/plugin&gt;
        &lt;/plugins&gt;

2) clover2:instrument + clover2:clover - it will not fail build in a parallel lifecycle, but fail in main cycle

&lt;plugins&gt;
            &lt;plugin&gt;
                &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                &lt;artifactId&gt;maven-surefire-plugin&lt;/artifactId&gt;
                &lt;configuration&gt;
                    &lt;testFailureIgnore&gt;false&lt;/testFailureIgnore&gt;
                &lt;/configuration&gt;
            &lt;/plugin&gt;
            &lt;plugin&gt;
                &lt;groupId&gt;com.atlassian.maven.plugins&lt;/groupId&gt;
                &lt;artifactId&gt;maven-clover2-plugin&lt;/artifactId&gt;
                &lt;version&gt;3.1.6&lt;/version&gt;
                &lt;configuration&gt;
                    &lt;licenseLocation&gt;${clover.licenseLocation}&lt;/licenseLocation&gt;
                    &lt;generateHtml&gt;true&lt;/generateHtml&gt;
                &lt;/configuration&gt;
                &lt;executions&gt;
                    &lt;execution&gt;
                        &lt;id&gt;report&lt;/id&gt;
                        &lt;phase&gt;validate&lt;/phase&gt;
                        &lt;goals&gt;
                            &lt;goal&gt;instrument&lt;/goal&gt;
                            &lt;goal&gt;clover&lt;/goal&gt;
                        &lt;/goals&gt;
                    &lt;/execution&gt;
                &lt;/executions&gt;
            &lt;/plugin&gt;
        &lt;/plugins&gt;

Execute using: mvn clean install

Thanks for the suggestions Marek. Currently, we only want to run Clover during site generation. So we run the "instrument-test" goal during the "pre-site" phase. We need to do "instrument-test" to avoid our Findbugs static analyzer failing the build analyzing the Clover instrumented source. How would you adapt the use of "setup" and "clover" to our situation?

As far as lifecycle.xml goes, it seems like a bad idea to alter the configuration of the Surefire plugin from the Clover2 plugin. Are you folks planning to change this "HACK" to a configuration parameter rather than a hard coded override?

Have you considered using profiles and running build twice: one with Clover and one with Findbugs?

We do use profiles and can consider separate builds but that is a lot of overhead for us for working around a bug in the Clover plugin. I would be more inclined to fork the plugin, apply a patch to fix the bug and use that in our builds.

Can I open a JIRA issue against the Clover2 plugin to address this bug and if so, could you provide the link?

I have created CLOV-1109 for this issue.

Suggest an answer

Log in or Sign up to answer
Atlassian Summit 2018

Meet the community IRL

Atlassian Summit is an excellent opportunity for in-person support, training, and networking.

Learn more
Community showcase
Posted 10 hours ago in Teamwork

What teamwork quotes inspire you?

Hey everyone! My name is Natalie and I'm an editor of the Atlassian Blog and I've got a question for you: What's your favorite quote about teamwork?  We've compiled a list here, along with...

27 views 3 4
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you