Why does the execution time to produce individual reports keep increasing?

[Tim Tapping]

Hi - We are using Clover Version 3.1.10, built on January 08 2013 (build-885).

Reports used to take 25 minutes to generate. Over time the duration has been increasing and become dependent on the numerical order when the reports are produced. That is to say, not only is the duration increasing in general, but 3rd report takes longer than the 1st, 45 minutes vs. 2.5 hours. We have 20 reports to generate and don't finish in 24 hours.

Example of console log out of the first report:

02:25:01  [clover-report] Loaded from: /data/jenkins-slave/workspace/codecov/src/buildtools/lib/clover-ant-3.1.10/lib/clover.jar
02:25:01  [clover-report] Loading coverage database from: '/data/jenkins-slave/workspace/codecov/src/.clover/clover3_1_10.db'
03:13:30  [clover-report] Writing HTML report to '/data/jenkins-slave/workspace/codecov/src/bin/BuildReports/udp_codecov'
03:15:11  [clover-report] Done. Processed 46 packages in 100606ms (2187ms per package).

Example of the 3rd report

06:54:46  [clover-report] Loaded from: /data/jenkins-slave/workspace/codecov/src/buildtools/lib/clover-ant-3.1.10/lib/clover.jar
06:54:46  [clover-report] Loading coverage database from: '/data/jenkins-slave/workspace/codecov/src/.clover/clover3_1_10.db'
09:23:46  [clover-report] Writing HTML report to '/data/jenkins-slave/workspace/codecov/src/bin/BuildReports/hoteldomain_codecov'
09:30:27  [clover-report] Done.

Background info:

Running on Linux with 32GB RAM

Remedial actions taken:

We were seeing out of memory errors so we split the Jenkins code coverage job into 2 parts:

1. trunk-unit-codecov-buildrun: has the ant targets 'clear-all' and 'unit-codecov'. The log shows compilation, the runs the unit tests and builds out clover.db
2. trunk-unit-codecov: only produces reports.
3. Increased RAM on the box from 16 GB to 32 GB

I have manually deleted the .clover folder, killed the jenkins process, restarted the box, restarted the jenkins process to no avail.

Has anyone seen this and (hopefully!) rectified the situation?

Thanks.

Tim

[Tim Tapping · Accepted solution]

Ruled out disk i/o issues by running: iostat -x 10 10

• Values were well within the norm

Ran 'top', found evidence that the CPU was getting hammered

Ran “ps aux | grep "java"” and noticed there were 4 instances of Clover running. Not good and the reason the box was CPU bound.

Major clean up was in order

• Deleted all instances of clover since there were some old versions, both in the SCC depot and on the machine.
• Downloaded the latest version and placed that in the SCC depot.
• Moved another jenkins slave job off the box.
• Bounced the box and restarted the jenkins slave
• Kicked off a new build

Ran “ps aux | grep "java"” and noticed there was 1 instance of Clover running

Now the Clover Reports are running as expected.

Not sure why 4 instances were spun up but keeping my eye on that

[Tim Tapping]

Additional info: The .clover db folder used to be 3 GB. What currently gets produced is 4.5 GB