code coverage of selenium IDE tests

[Shankar KC]

Hi we have a bunch of legacy selenium IDE tests for our product old feature. Is it possible to get code coverage of these selenium IDE scripts? They are NOT JUnits. I want to use clover test optimization for our build system to add the JUnits and selenium IDE scripts to minimize the test execution time and run the tests that actually matters for the code change. Usually we run selenium IDE tests from command prompt as below :-

java -jar D:\SeleniumTest\selenium-server-standalone-2.33.0.jar -htmlSuite "*firefox" "http://www.google.com" "D:\SeleniumTest\SeleniumSuite.HTML" "D:\SeleniumTest\seleniumtestresult.html"

Thanks for your help

Shankar KC

[Marek Parfianowicz]

Hi Shankar,

Yes, it's possible with Clover, however it would require effort on your side in order to change how your selenium tests are launched.

In this test configuration, several processes are running:

JVM process with selenium driver ---> Firefox process ---> JVM process with your application backend

I assume that you want to measure per-test coverage for selenium tests and optimize them according to classes called at your application's backend.

First thing is to enable a distributed coverage feature in Clover:

https://confluence.atlassian.com/display/CLOVER/Working+with+Distributed+Applications

Second thing is to tell Clover when your Selenium test starts and ends. This is tricky, because there's no way to do this at runtime (there's a feature request for it - CLOV-127 - but it's not planned in a near future). Clover can identify test boundaries at compile time only - it recognizes JUnit and TestNG frameworks by default.

It means that you have to wrap call of every single Selenium test inside a single JUnit/TestNG test method and instrument such test wrapper with Clover. Such wrapper could look like this:

public class MySeleniumTests {
   @Before
   public void setUp() {
      // call System.exec() with Selenium runner, runner should pause
   }
   @Test
   public void seleniumTest1() {
      // send signal to run test #1
   }
   @Test
   public void seleniumTest2() {
      // send signal to run test #2
   }
   ...
}

Next you have to run MySeleniumTests with -Dclover.server=true JVM property.

The trick works this way:

• Clover instruments MySeleniumTests class and adds special instrumentation for every test method, which will record "test start" / "test end" events
• Clover instruments both MySeleniumTests class and your entire application code with a Distributed Coverage feature enabled; it means that Clover's coverage recorders running in different JVMs will communicate with each other in order to record per-test coverage
• You run MySeleniumTests class with -Dclover.server=true; it starts the "Clover Server" in this JVM
• JUnit runner calls MySeleniumTests.setUp() which starts the selenium runner process; but tests shall not start yet! (it means that you'd have to modify your selenium tests)
• JUnit calls MySeleniumTests' test methods; each of them informs selenium runner to run a single selenium test
• Clover in the meanwhile detects test boundaries and communicates with an application's backend and sends "test start" / "test end" events to other Clover's coverage recorders

The problem is how to:

• pause selenium runner
• send signals to run testX

It's up to you how to implement this. For instance, your selenium tests could wait until some marker file becomes available on a disk; and such file would be created in MySeleniumTests class. Or your could read data from a pipe. Etc.

See more details here:

https://confluence.atlassian.com/display/CLOVER/Measuring+per-test+coverage+for+manual+tests

Cheers
Marek