Fisheye/Crucible becomes unusably slow over time (24-48 hrs) and consumes lots of CPU

[Oliver Meyn]

Initially our installation was fine (starting around 2.7-ish) and then towards the end of 2.8 we started seeing dramatic slowdowns after the install had been running for a few hours/days (typically 24-48 hrs). When slow, and even without any users touching the app, top says the CPU is using at least 100%, often more. Restarting the app gets it back to normal speed.

It is not heavily used - we only have 8 people with accounts, and the slowdown happens when even a single reviewer is doing a review. It does seem to be related to reviews - browsing around doesn't seem to cause the problem. Though could be that we only care about performance when trying to do review so only notice then.

It has 4gb heap, and the VM it lives in has 16gb ram and 8 cores. It shares the VM with Jira and Confluence and the postgres instance they all read from, but that's it. Our svn repos are on googlecode. I upped the # of threads for indexing as suggested in your fecru performance tuning page.

As of issue creation it is in its slowdown state, publicly accessible at: http://dev.gbif.org/code

[Sergey Svishchev]

Try to find out which thread(s) are consuming the CPU and look at thread dump to find what are they doing (use JDK's jstack to take the dump). If GC logging is enabled, look there for signs of thrashing too

[JamieA]

I see similar problems with 2.9... it grinds to a halt every 2 or 3 days.