Clover history / threshold ratcheting seems inconsistent.

[Brett Cave]

hi,

I have a project that I build with Gradle, using the ant tasks for Clover to check coverage. As part of the setup, I have configured history points and a threshold of 0 - i.e. coverage cannot decrease between builds.

The project is a multi-module project. The build fails when source code from 1 of the other modules changes - i.e. there is no change in coverage what so ever. Test & coverage is only configured for 1 of the modules.

here is how clover is configured to run (with gradle):

ant."clover-clean"()
ant."clover-setup"(initString: "$buildDir/clover/test.db", tmpDir: $buildDir/clover/tmp") {
  ant.fileset(dir: "src/main/java")
  ant.fileset(dir: "src/test/java")
}

// move classes, compile instrumented source.

// Run tests

ant."clover-check"(threshold:"0%", historydir:"history")
ant."clover-report" { // report definition }
ant."clover-historypoint"((historyDir:"history")

line 11 above checks the current coverage against history points. While testing this, I removed all history points except for one, made a change to a module that does not run tests, and re-ran the test with coverage. The coverage fails.

When coverage fails, new history points are not saved, i.e. the clover-check on a subsequent build gets re-run against the same checkpoint - and the summary shows completely different values for changes in coverage - really inconsistent. Any ideas as to why this is happening?

[Brett Cave · Accepted solution]

clover was in fact working correctly. Our in-container test cases (arquillian) reference data sources with varying data, and different conditions trigger different processing - so our application runs different code based on what data is returned.

we have set the threshold to a value that takes this into consideration, and are looking at updating test cases to wok with a predictable data set.

[Brett Cave]

Definitely seems to be a bug somewhere.

I ran the gradle task that invokes clover via ant now, it passes and generates a history point. I then commit 1 change to another module, that has no testing in it configured at all (the change I make is a wording change on a UI component, so absolutely no impact on coverage), and then re-run test with coverage enabled:

Total coverage of 44.26% did not meet last history point target of 44.28%
  -22.2 to 60% com.package.web.goal.SomeGoal
  -2.9 to 58% com.package.util.ChartingPeriodUtil

Clover is configured to fail the build if the threshold of 0 is not met. i.e. the failed coverage build DOES NOT record another history point.

Then, within my CI system, I re-run the task, without any change to source at all, and it passes, i.e. the threshold passes.

Any idea why this changes, or what info I need to provide in order to determine this?