Clover Per Test Coverage for Automated Functional Test

My product code and Automation code are in totally different projects. We use TestNG for automation. 

Currently we instrument product code using clover and then run our automation to generate coverage report. This gives the overall number, but i would like to know if it is possible to generate per test coverage report in my case? 

If yes, how do i do it ? 

Thanks !

2 answers

1 accepted

0 vote

No, Build is create on different machine and deployed on different machine. Also Test code and Application code are built separately. Application is deployed on one system and tests are executed from different system.

In such case the scenario will look slightly different:

1) Instrument your application code and test code. I assume that each has only one database produced. Use "interval" or "threaded" flush policy.

  • two Clover databases are created, let's name them "clover-test.db" and "clover-app.db"

 

2) Copy application and test classes/JARs to your test machines. Copy "clover-test.db" and "clover-app.db" databases as well. You can put them in any location, just ensure that JVM processes will have a write access to directories.

 

3) Start your tests with the following JVM options:

-Dclover.distributed.coverage=numClients=1 -Dclover.server=true -Dclover.initstring=/path/to/clover-test.db
  • numClients=1 will ensure that test execution will pause until your application is started, see next point
  • clover.initstring points to a database on your test machine

 

4) Start your application with the following JVM options:

-Dclover.distributed.coverage=host=your.test.server -Dclover.initstring=/path/to/clover-app.db
  • host is a name or IP of your test server where tests are launched
  • clover.initstring points to a database on your test machine
  • as soon as application is started, instrumented code will connect to the "Clover Server" and test execution will proceed

 

5) As soon as tests are finished, you shall see coverage recording files for both databases on both test machines. Copy coverage recording files (clover-test.db* and clover-app.db*) back to your build machine - they will be used for report generation. Put these coverage recording files in the same directory(-ies) where databases are located.

 

6) Merge two Clover databases. You can use <clover-merge> (Ant), clover2:aggregate or clover2:merge (Maven) or CloverMerge (command line tool) for this purpose. Example for Ant:

&lt;clover-merge initString="clover-merged.db"&gt;
    &lt;cloverDb initString="clover-app.db"/&gt;
    &lt;cloverDb initString="clover-test.db"/&gt;
 &lt;/clover-merge&gt;

 

7) Use a merged database to generate HTML report. Example for Ant:

&lt;clover-report initstring="clover-merged.db"&gt;
   &lt;current outfile="clover-html"&gt;
      &lt;format type="html"/&gt;
   &lt;/current&gt;
&lt;/clover-report&gt;

 

Feel free to ask in case you get stuck in any point.

 

Cheers
Marek

Hi Marek, Followed above mentioned steps, but when i generate report it does not show me any test coverage. Test Code tab shows 0% coverage and I get below message on Test Results tab. "No test results No test results could be found. Please ensure that you have instrumented your unit tests correctly." My application and Automation projects are different and are instrumented separately to get two clover.db's I also noticed below error while generating the report: [clover-report] Ignoring unknown context filter "hashCode" [clover-report] Ignoring unknown context filter "equals" [clover-report] Ignoring unknown context filter "toString" Thanks

Do you see both global coverage recording files (they're named like 'clover.dbhash_code' and 'clover.dbhash_code.1') and per-test coverage recording files (they're named like 'clover.dblonger_hash_code.s') after test execution? If not, then please check your application logs, the most probably there'll be a message saying that Clover was unable to find the database. If such message is present - please check if -Dclover.inistring property has been passed. In case you have coverage recording files - did you copy them back to your build server? If yes, then what messages do you see in <clover-merge> or <clover-report>? There shall be information about number of recording files processed. "Ignoring unknown context filter" - did you use the same set of context filters for both application code and test code? If yes, then does your <clover-report> use context filters which have been defined in <clover-setup>?

4) Start your application with the following JVM options: -Dclover.distributed.coverage=host=your.test.server Did you use name of the machine where functional tests are running?

Do you see any Clover's log messages related with a distributed coverage feature in both application and test logs? I'm asking about messages like: "Distributed coverage is enabled with: <settings>" "Starting distributed coverage service." "Starting distributed coverage client: <settings>" "Clover waiting for 1 remote clients to attach to this remote testing session. " "Could not initialise Distributed Coverage collection in Clover: <exception>" ...

- Coverage recording files are getting created on Application Machine as well as on Test Machine - On Test machine I am using below options to start the tests -Dclover.initstring=path/to/cloverdb/clover-test.db -Dclover.distributed.coverage=host=<hostname of my test machine>;numClients=1 -Dclover.server=true And I get following message : Distributed coverage is enabled with: name=clover.tcp.server;host=<my test machinename>;port=1198;timeout=5000;numClients=1;retryPeriod=1000 Starting distributed coverage service. Clover waiting for 1 remote clients to attach to this remote testing session. - On Application I am using options -Dclover.initstring=path\to\clover.db -Dclover.distributed.coverage=host=<Host name of TestMachine> - On Merge I get following output : [clover-merge] Merging database 1 of 2: clover-test.db [clover-merge] Merging database 2 of 2: clover.db [clover-merge] Writing merged database registry [clover-merge] Merge complete - After running the report on merged db I do get following message [clover-report] Done. Processed 459 packages in 87405ms (190ms per package).

- Coverage recording files are getting created on Application Machine as well as on Test Machine Did you copy them back to a build machine? Could you please run <clover-merge> task with debug logging (ant -d) and find messages similar to these: "Processed NN recording files in MMms ..." "Processed NN per-test recording files in MMms" and compare number of coverage recording files from these log messages with a number of files present on a disk? Could you also check if there are any messages like these: "Ignoring coverage recording <file name> because no FileInfo supports its coverage range"

To merge / generate report, i have a separate folder in which i copy the all the files from test and application and then run the command to get report or merge the db's. I can see below lines when run in debug mode [clover-merge] Ignoring coverage recording FileBasedCoverageRecordingTranscript[header=Header[dbVersion=1426360953414, writeTimeStamp=1426367764985, format=0], coverageSum=17573, hitCounts.length=124683] because no FileInfo supports its coverage range [clover-merge] Processed 0 recording files in 11ms (- per recording) [clover-merge] Processed 0 per-test recording files in 0ms (- per recording)

Now i don't see any "Ignoring coverage recording .." message. Also merge log shows below lines [clover-merge] Processed 3 recording files in 19ms (6ms per recording) [clover-merge] Processed 0 per-test recording files in 0ms (- per recording) But now when i generate the report, my automation test project is listed as part of Application Code, And Test Results shows "No Test Results" Why there is 0 per-test recording ? even after files are being processed.

The most probably Clover did not classify your tests as test classes. Clover recognizes JUnit, TestNG, Spring and Instinct annotations as follows: https://confluence.atlassian.com/display/CLOVER/Clover+test+detection Please check whether naming convention of your classes and annotations used match any of these patterns. If not - you can "teach" Clover how your tests are named - see the <testsources> tag in <clover-setup> (https://confluence.atlassian.com/display/CLOVER/clover-setup). Note: in case you use <testsources> tag in <clover-setup>, you will have to use the same <testsources> tag in <clover-report> too.

> Why there is 0 per-test recording ? even after files are being processed. See my comment above. You can also run build with debug logging (ant -d) and see output produced by <clover-setup>. In case it finds any test method/class, Clover will print messages like the following: [clover] Clover all over. Instrumented 2 files (1 package). [clover] 23 test methods detected.

When i instrument my AutomationProject, i do see following log message, [INFO] Clover all over. Instrumented 219 files (68 packages). [INFO] 884 test methods detected. do I still need to use <testsources> ?

I don't know how many tests do you have. But number != 0 suggests that Clover recognized your test sources correctly. At least in the <clover-setup> task. Could you make a test and define <testsources> tag in <clover-report> to see what happens?

Hi Marek, Tried adding testsoruces tag but still no success. It shows the tests project as application code. Once again to brief the steps i am following are : 1. Instrument ApplicationCode, this generates clover.db 2. Instrument AutomationCode, this generates clover-tests.db 3. Copy clover.db and clover-tests.db to a shared n/w path. 4. Copy instrumented application jars to application machine and Start application with -Dclover.distributed.coverage=host=your.test.server -Dclover.initstring=/shared-network/path/to/clover.db 5. Copy the instrumented Automation jars to test machine and Run Automation Code from Test Machine with -Dclover.distributed.coverage=numClients=1 -Dclover.server=true -Dclover.initstring=/shared-network/path/to/clover-test.db NOTE : Both the clover.db and clover-test.db are in shared path and accessible to both test and application machine. 6. Once the test run is complete, i do see clover.db* and clover-tests.db* files in shared folder. 7. Then i copy all the files from shared folder to a new folder on build machine. 8. And run ant to merge both the db's. I am using following in my build.xml <project name="clover" default="clover.html" basedir="."> <taskdef resource="cloverlib.xml" classpath="${basedir}/clover-4.0.2.jar"/> <target name="clover.html" description="Generates clover report"> <clover-merge initString="clover-merged.db"> <cloverDb initString="clover-test.db"/> <cloverDb initString="clover.db"/> </clover-merge> </target> </project> 9. Then on the same folder i run clover report using below build.xml <project name="clover" default="clover.html" basedir="."> <taskdef resource="cloverlib.xml" classpath="${basedir}/clover-4.0.2.jar"/> <target name="clover.html" description="Generates clover report"> <!-- Generate HTML report --> <clover-report initstring="C:\Clover\target\clover-merged.db"> <current outfile="clover-html"> <format type="html"/> </current> </clover-report> </target> </project> It does generate report but shows no test results, instead my automation project is also listed in application.

Hi Marek, its working, seems like test classes should end with *Test*.java only then it adds it as test code instead of Application Code.

Clover's test detector recognizes a class as the test one when any of the following conditions are met: * class has @org.testng.annotations.Test or @org.testng.annotations.ExpectedExceptions annotations * class has a javadoc @testng.test tag * class has 'test' in its name (case insensitive) * super class is TestCase or junit.framework.TestCase or has 'test' in its name This test detector can be overridden, as I mentioned before.

0 vote

Hi Aryan,

The exact answer depends on how your development environment is configured. Optimal Clover configuration can depend on a number of factors, such as:

  • build separation - do you compile application code and test code separately?
  • JVM separation - do you run test code in one JVM and an application runs in another JVM?
  • machine separation - do you build and test using the same physical machine? or maybe application runs on a dedicated test server?

 

 

I would say that in a "standard" configuration:

  • application under test is deployed to some application server (Tomcat / JBoss / ...)
  • compilation and tests are performed on the same physical machine
  • tests are launched from a build (e.g. "mvn test")

In such case you would have to do the following:

1) Instrument your application code and test code. Use absolute paths to a database in "initstring". Use "interval" or "threaded" flush policy.

  • thanks to this you will have no problems with running application/test code from a different working directory
  • thanks to this you will have two separate Clover databases - one for application and one for tests
  • thanks to this coverage data will be recorded on disk on the fly, without waiting for JVM termination

2) Start your application with the following JVM option:

-Dclover.distributed.coverage=ON
  • thanks to this the Distributed Per-Test Coverage feature will be enabled
  • application will start as "Clover Client", you may see connection attempts to the "Clover Server" - see below

3) Start your tests with the following JVM options:

-Dclover.distributed.coverage=ON -Dclover.server=true

4) As soon as tests are finished:

  • you shall see coverage recording files along with both databases

5) Merge two Clover databases. You can use <clover-merge> (Ant), clover2:aggregate or clover2:merge (Maven) or CloverMerge (command line tool) for this purpose. 

6) Use a merged database to generate HTML report.

  • if everything went well, you shall see not only global coverage but also per-test coverage in the report

 

You can find more hints here:

 

warning ad 3&4) You may encounter some issues related with starting "Clover Server" and "Clover Client". In case of time-outs or starting them in a wrong order you may not find per-test coverage in a report. In such case, please have a look at hints in the "Working with Distributed Applications" manual. Quite often it's necessary to start tests with "-Dclover.distributed.coverage=numClients=1" option.

 

Feel free to ask if you have any questions.

 

Cheers
Marek

Suggest an answer

Log in or Join to answer
Community showcase
Emilee Spencer
Published 2 hours ago in Marketplace Apps

Marketplace Spotlight: DeepAffects

Hello Atlassian Community! My name is Emilee, and I’m a Product Marketing Manager for the Marketplace team. Starting with this post, I'm kicking off a monthly series of Spotlights to highlight Ma...

22 views 0 3
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you
Atlassian Team Tour

Join us on the Team Tour

We're bringing product updates and pro tips on teamwork to ten cities around the world.

Save your spot