Per-test-coverage does not report all test cases

[kvallish]

I have an application war that is instrumented via clover and deployed on tomcat on a standalone server. The project is multi-module with structure as below:

main project pom

|---module A (client related)

|---module B (rest related)

|---module C (database related)

|---module D (depends on all the above for successful compilation)

Module D is the one that generates the application WAR file and is the project that is configured for clover instrumentation. This is the main module that has most of the application logic and hence needs to be covered for per-test coverage. The plugin configuration that I have used to instrument the source is as below:

<flushPolicy>interval</flushPolicy>
<flushInterval>500</flushInterval>
<singleCloverDatabase>true</singleCloverDatabase>
<instrumentation>method</instrumentation>
<includesAllSourceRoots>true</includesAllSourceRoots>
<distributedCoverage />

The application is a java web application launching inside tomcat, hence I have passed the below arguments to the JVM

-Dclover.logging.level=debug \
-Dclover.distributed.coverage=host=<ip where my tests are running> \
-Dclover.initstring=/path/to/webapps/clover.db \
-Dclover.flush.everytest=true \

The end-to-end tests are run from another test box which acts as clover-server. I launch my tests with the below command:

mvn clean clover:setup test -Dtest=SampleClass#sampleTestMethod -Dclover.server=true -Dmaven.clover.cloverDatabase=./clover_tests.db -Dclover.initstring=/absolute/path/to/clover_tests.db

In this configuration am unable to see per-test-coverage results for all the tests that are run.  If I try another test class in the above maven command, I do get the per-test coverage correctly. Visibly, there are no striking differences between the two test classes and they seem to test similar areas of the product.

Most of the product code where a hit is expected shows "No tests hitting this source file were found". All executed test cases do show up as executed only under "Test Code" tab. Since some of the tests do show up, am wondering anything wrong with my setup or configuration.

NOTE: During test code instrumentation, I do see the below warning multiple times. Not sure if it is related to the issue am facing, but I do not have any CLOVER:OFF statements in my test code:

[WARNING] Trying to add a statement but current method is null. Did you put CLOVER:OFF before a method signature and CLOVER:ON inside a method body?

Any thoughts on what I could be missing?

[Marek Parfianowicz · Accepted solution]

First, please verify if your tests are set up correctly:

1. Check if all your tests classes have been instrumented. If they were, you shall see them in the HTML report on the 'Test code' tab. If not - check inclusion/exclusion patterns, check if Clover was set up for given module, check if instrumentation was executed before compilation etc.

2. Check if global code coverage have been recorded for test classes. In the HTML report they have green highlight in their source code. If not - check if clover.db file was accessible at test execution time (there shall be warnings in the console), check if you really executed instrumented classes and not the normal ones (e.g. 'clover:setup test clean test' would wipe-out instrumented code and re-run tests for normal code).

3. Check if per-test code coverage have  been recorded for test classes. In the HTML report if you click on a test method (left margin with line numbers) you shall see "Line was covered by tests" dialog, i.e. test should cover itself. If you don't see it - please check if the test framework used is supported by Clover, if not - ensure you've got correct test patterns set (Clover-for-Ant).

Next, please verify if your application under test was set up correctly:

4. Check if all your application classes have been instrumented. You shall find them on the 'Application code' tab.

5. Check if global code coverage have been recorded. Again, look at the HTML report. If not - ensure that clover.db was available at runtime. Use -Dclover.initstring or -Dclover.initstring.basedir to point to the correct location of the files. 

6. Check if per-test coverage have been recorded. If it is missing for some tests - ensure that application under tests started *before* test execution. This is relevant for multi-JVM tests (e.g. tests from one JVM calling application on a web server running in another JVM). Use 'clover.distributed.coverage' and numClients parameter.

Lastly, please check if test results have been collected correctly:

7. It may happen that test errors collected by Clover (exceptions thrown from tests executed) do not match test results from TEST-*.xml files. There may be few reasons: 

* JUnit test annotations (e.g. @techoneway with excepted exception) are not recognised

* test execution and verification may be separated into different phases - it's a case for maven-failsafe-plugin - tests run in integration-test phase, validation happens in post-integration-test phase.

In such case, configure the report target (<clover-report> for Ant, clover:clover for Maven) to use test results from TEST-*.xml files.

[kvallish]

1,2,3 were perfectly fine.

Regarding 4, there is a glitch. The application under test is built in 2 steps:

1. The core(aka common) project is built first. This is a separate repo and maven project that has 20 sub-modules. The output of each of the modules is a jar file. This is not instrumented
2. The main project is built next. This has around 4 modules and it consumes the modules/jars generated in previous step. The output of the last module in this project is a WAR file that has the jars of the previous step packaged into the WAR. This project is instrumented and clover.db is also generated.

The main reason that step 1 is not instrumented (though I wanted it to be instrumented) is because the project is separately built and would have generated a separate clover.db file. Since the application is started using tomcat with exactly one WAR file, I have no way to give more that one -Dclover.initstring param during start-up. I tried giving -Dclover.initstring.basedir to point to the root where 2 clover.db files are kept, but the application failed to start stating that it cannot find clover.db file.

5,6,7 were all cross checked and they are looking fine. 

Thank you for the inputs. Any suggestions for the above predicament?

[Marek Parfianowicz]

If your application is being executed on the same machine where compilation is performed then it shall be no problem to have more than one clover.db file. Or more precisely - if the folder layout is the same. Instrumented classes "remember" full path to their clover.db.

It means that you can have one application using multiple clover.db files.

If folder layout is different then it shall be no problem too. What you can do is that you can share clover.db file between compilations (it can be updated by appending new instrumentation session). 

So you could build your core, create clover.db file, save it as an artifact.

Next you could build rest of your application, using the clover.db artifact from the previous step. It will be updated and information about new classes added.

Next use the updated clover.db for application & test execution. 

[Marek Parfianowicz]

If you want to use clover.inistring.basedir, then you have to use relative paths to clover.db in your modules. E.g.

```

application  - base dir = application

   /core

     /target/clover.db - initstring=core/target/clover.db

   /main

     /target/clover.db - initstring=main/target/clover.db

```

and then when executing tests (also on another machine):

```

workdir - copy clover.db files preserving relative paths

   /core/target/clover.db

   /main/target/clover.db

```

Tell what the root folder is:

-Dclover.initstring.basedir=workdir

[kvallish]

I was able to now get recording files for both the core and the main applications. 

But the original problem remains. For few test cases I am able to get the test code<->product code mapping in the report. For few other test cases am unable to. If the non-working test case code is copy/pasted in the test method that was working, the pasted method works correctly. 

Totally clueless on why one test works while other does not. Both tests have @techoneway annotation.

[Marek Parfianowicz]

I suspect that the problem lies in the way how you create an application WAR and deploy it and how do you later execute tests against the application. Probably you run a "mvn clean" command in between and as a result you the clover.db is wiped out. Is it something like this?

mvn package # to create a WAR
scp ...     # to deploy a WAR and clover.db
mvn clean clover:setup test ... # to run tests

Please try to perform entire instrumentation without a cleanup in between. So something like this:

mvn clean clover:setup package ... # to instrument app & tests and create a WAR
scp ...   # to deploy a WAR and clover.db
mvn test ... # to execute tests, without clean, without recompilation

[Marek Parfianowicz]

Regarding:

The main reason that step 1 is not instrumented (though I wanted it to be instrumented) is because the project is separately built and would have generated a separate clover.db file.

You can configure a different initstring for application and tests, e.g.:

<initstring>clover-app.db</initstring>
<initstring>clover-tests.db</initstring>

As these have just a file name, it will be easier for you to copy them into one directory and use -Dclover.initstring.basedir parameter.

[kvallish]

At last I was able to figure this out. The classes that showed up in the per-test coverage report have "Test" in the class name whereas the ones that did not show up did not have the word "Test" in the class name.

@Marek Parfianowicz , 

Is this by design? Is there a way I can tell clover to consider "all" class names as test classes regardless of their naming pattern?

[Marek Parfianowicz]

Yes. You can specify naming patterns for tests, see:

Ant - <clover-setup> / <testsources>

Maven - clover:setup / <testSources>

[kvallish]

Ok.

Our tests do not have any class or method naming conventions. So, to workaround the problem, I added the below:

<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <version>4.4.1</version>
    <configuration>
        <distributedCoverage>
            <host>10.10.10.10</host>
            <port>1198</port>
            <numClients>1</numClients>
            <timeout>10000</timeout>
        </distributedCoverage>
        <testSources>
            <testClasses>
                <testClass>
                    <name>*.java</name>
                </testClass>
            </testClasses>
        </testSources>
    </configuration>
</plugin>

Would this not consider ALL classes as test classes while reporting per-test coverage? Am unable to see the test classes in the report unless the classes are appended with "Test" in its name.

Can you suggest the right configuration such that all test classes (they reside under src/test/java) and test methods are considered irrespective of their names?

[Marek Parfianowicz]

@kvallish Please have a look at the example in http://openclover.org/doc/maven/latest/setup-mojo.html#testSources:

<testClass><name> contains a regular expression (so not Ant-style * or ** asterisks) of the class name (and not the file name). 

Please try:

<testClass><name>.*</name></testClass>

[kvallish]

Wow, that worked!

Thanks a lot. Closing this thread.

Edit:

Oops, I closed it too soon. Now what I see is the report has beforeMethod and afterMethod reported as well. To filter out test methods, I added testMethods tag. My complete configuration looks like below:

<testSources>
    <testClasses>
        <testClass> <!-- 0..N occurrences -->
            <name>.*</name>
            <testMethods> <!-- 0..N occurrences -->
                <testMethod>
                    <name>.*</name>
                    <annotation>@Test</annotation>
                </testMethod>
            </testMethods>
        </testClass>
    </testClasses>
</testSources>

Now again, there are no per-test mappings.  Am I missing something @Marek Parfianowicz ?

[Marek Parfianowicz]

There may be a bug in documentation. Could you please try without '@' character?

<annotation>Test</annotation>

[kvallish]

Yes, using the below detected the exact classes and methods without having to rename the test classes or test methods:

<testSources>
    <testClasses>
        <testClass> <!-- 0..N occurrences -->
            <name>.*</name>
            <testMethods> <!-- 0..N occurrences -->
                <testMethod>
                    <name>.*</name>
                    <annotation>Test</annotation>
                </testMethod>
            </testMethods>
        </testClass>
    </testClasses>
</testSources>