Clover mapping for manual test cases

[Istvan Viczian]

We would like to use the Atlassian Clover for manual testing. It works like a charm, but we would like to see which manual test case covers which source file.

It would be a test optimalization for manual test cases.

It would be very useful to know, because after this we could run only the affected manual test cases again (the manual resource is more expensive then the resource of the ci server :). It would be a mapping between manual test cases and source files.

It is available for JUnit test cases, and we can extract, see the link:

https://answers.atlassian.com/questions/93934/is-there-a-supported-way-to-extract-the-source-file-to-relevant-tests-mapping-from-a-clover-snapshot-file

Unfortunately it does not appear in the XML report, it's another feature request. :)

An API would be the best, where we could instruct Clover (for example to pass an id). It would be perfect for any automatic testing tool, for examle Selenium, to give a signal to Clover about test "context". (For example the id of the Selenium test case.)

Is this API exists, or could you give a workaround for this problem?

[Marek Parfianowicz · Accepted solution]

Hi Istvan,

Great question!!

Yes, it is possible to "hack" Clover and measure per-test coverage from manual test cases too. The only thing which has to be done is to "tell" Clover when manual test case starts and ends. The easiest way to provide this information is to write a JUnit test case, with one test method for each manual test case. Clover will add "start test / end test" instrumentation to such methods. Next you have to run this JUnit test together with your application - I suggest using Distributed Coverage feature for this as you will be able to start your application as usual.

Steps:

1) Write JUnit test case having following features:

• tests do not start until application under test is launched
• one test method per one manual test case
• single test method starts just before corresponding manual test case is started
• single test method ends just after corresponding manual test case is finishedto

Example:

import junit.framework.TestCase;

public class MyManualTest extends TestCase {

    public static void main(String args[]) {
        MyManualTest myTest = new MyManualTest();
        myTest.waitUntilYourApplicationStarts();
        int testNo = myTest.getTestNumber();
        switch (testNo) {
            case 1:
                myTest.testManualTest1();
                break;
            case 2:
                myTest.testManualTest2();
                break;
        }
    }

    private int getTestNumber() {
        // e.g. entered by user / read from commandline arg ...
        return 1;
    }

    private void waitUntilYourApplicationStarts() {
        // e.g. "Press any key when ready" / check for existence of some marker file
    }

    private void waitUntilTestEnds() {
        // e.g. "Press any key when test is finished" / check for some marker file
    }

    public void testManualTest1() {
        // Clover will add "test start" here
        waitUntilTestEnds();
        // Clover will add "test end" here
    }

    public void testManualTest2() {
        // Clover will add "test start" here
        waitUntilTestEnds();
        // Clover will add "test end" here
    }

}

2) Compile JUnit test case together with your application under test (produce two jars, for instance) with Distributed Coverage feature enabled.

• flushpolicy = interval or threaded might be needed

Example:

<clover-setup initstring="/path/to/my/clover.db" flushpolicy="interval" flushinterval="1000">
    <distributedCoverage/>
</clover-setup>

3) Run unit test and application:

a) unit test shall be launched with -Dclover.server=true parameter, e.g.

java -cp ... -Dclover.server=true MyManualTest

b) application shall be launched as usual (note that distributed coverage configuration is already compiled into instrumented classes), e.g.

java -cp ... MyApplication

Regards

Marek

[Marek Parfianowicz]

As soon as your tests are finished, you can generate HTML report in which you can track code coverage down to a single source line:

• open report page for a single class
• click on "Show Tests" link (just below the summary header); a list of all releated test cases will be shown
• select test case(s) you're interested in; lines which were hit by selected test case(s) will be highlighted

You could also generate test optimization snapshot file and use SnapshotPrinter to get the "source file-test case" mapping.

[Istvan Viczian]

It's a brilliant solution! We will try it in the trial period. Thank you very much!