Per-test coverage of integration tests with maven-failsafe-plugin

dmitry-popov September 26, 2017

Hello!

I'm struggling with setting up per-test coverage for integration tests.

I have the simple setup:

  • single clover database
  • single application
  • Maven 3 build
  • all tests and servers are run on localhost
  • integration tests are run by starting WildFly with Arquillian, clover-runtime.jar is added to the deployed artifact. Integration tests are run by maven-failsafe-plugin
  • I run
    mvn clean clover:setup install 
    to build the project and run the tests.
  • The i run
    mvn clover:clover
    to generate the html report based on clover.db.

For unit-tests, per-test coverage works superbly: in the html-report on covered lines I see a green line, and by clicking on it it displays the tests that cover this line.

For integration tests, the green/red coverage is shown, but there is no green line and I can't see, which lines are covered by tests.

I tried to do instrumentation-level distributed coverage by adding

<distributedCoverage/>

to clover-maven-plugin configuration
and

<configuration>
<forkMode>once</forkMode>
<systemProperties>
<property>
<name>clover.server</name>
<value>true</value>
</property>
</systemProperties>
</configuration>

to maven-surefire-plugin configuration, according to Using Distributed Per-test Coverage page.

But nothing changes: I still do not see test coverage on classes run on the server.
Logs about running distributed coverage are written to maven log.

Running TestSuite
Distributed coverage is enabled with: name=clover.tcp.server;host=localhost;port=1198;timeout=5000;numClients=0;retryPeriod=1000
Starting distributed coverage client: name=clover.tcp.server;host=localhost;port=1198;timeout=5000;numClients=0;retryPeriod=1000

The strange thing is that when I look to WildFly log, I see similar logs too (after server start):

2017-09-26 17:35:32,895 INFO  [stdout] Distributed coverage is enabled with: name=clover.tcp.server;host=localhost;port=1198;timeout=5000;numClients=0;retryPeriod=1000

2017-09-26 17:35:32,896 INFO  [stdout] Starting distributed coverage client: name=clover.tcp.server;host=localhost;port=1198;timeout=5000;numClients=0;retryPeriod=1000

which makes me think that the remote server behaves as Clover server, not as the client.


Questions:

1) Do I have to use distributed coverage at all, since the decision matrix says that for my simple case I just have to use "showUniqueCoverage=true"? I tried to add this to maven configuration, but this does not work as well.

2) Since integration tests are run via maven-failsafe-plugin, I suppose we have to add some configuration to this plugin too.
I've seen this page, but it looks extremely complex and obsolete with aggregate reports and huge report descriptors.
How to configure maven-failsafe-plugin correctly?

3) How do I understand that distributed coverage has started successfully both for the server and client sides?

4) How to make my WildFly server work as Clover-client, not as Clover-server?

 

Thanks in advance for your answers!

Regards, DP.

1 answer

1 vote
Grzegorz Lewandowski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 27, 2017

Hey,

Starting distributed coverage client

Suggests Clover started in a client mode, not server one. Therefore I think there's some problem in initialisation on a tests side.  When Clover starts in a server mode you'll see something like that:

Starting distributed coverage service.

Second thing with concerns me is numClients property I think it should be 1 since you have single application server. Without that, Clover tests won't wait for a clients to start.

If you have 2 JVM's, one where tests are executed and second for application server then I believe you need distributed coverage in order to get per-test coverage data. 

 

The page you refer is about reports. First of all, make sure you invoke instrumented integration tests (by any means, even taking a peek at tests bytecode) 

 

Cheers

dmitry-popov September 27, 2017

Hello, Grzegorz.

Starting distributed coverage client

Suggests Clover started in a client mode, not server one.

Oh, really, I was pretty dumb. So the server-side log is correct and it is starting in client mode.

Therefore I think there's some problem in initialisation on a tests side.

Yes, thanks for noting this. So we see, that my maven-test-side is starting in client mode. This strongly matches with my concern about running integration tests via maven-failsafe-plugin while configuration is added to maven-surefire-plugin. Do I have to add the same configuration to maven-failsafe-plugin? I'll have a try.

I also have to mention that even for unit-tests it seems that i can't telnet localhost 1198 So possibly even maven-failsafe-plugin config is not working :(

Second thing with concerns me is numClients property I think it should be 1 since you have single application server. Without that, Clover tests won't wait for a clients to start.

I don't think this can happen since Arquillian-test wait for WildFly to start. So I'm pretty sure that is not the case since the tests itself pass successfully.

I'll try with numClients = 1, anyway.

Will I see any logs on client-side or server-side when Clover-Client connects to Clover-Server?

If you have 2 JVM's, one where tests are executed and second for application server then I believe you need distributed coverage in order to get per-test coverage data. 

Thank you, at least now I know that it is impossible to do per-test-coverage for different JVMs without the distributed coverage.

First of all, make sure you invoke instrumented integration tests (by any means, even taking a peek at tests bytecode) 

I'm pretty confident that my tests are instrumented.
I see the instrumented sources in target\clover\src-test-instrumented.
I see instrumented classes in \target\test-classes.
I do not use includesTestSourceRoots = false or anything similar.
And, finally, I see the test classes in clover-coverage-html-report.

 

Thanks for answering.

Regards, DP.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events