nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

how to diagnose why crucible is slow

Paul Wagland

This is a simple enough question, but it probably has a complicated answer.

Given that we have a Crucible installation, running against subversion, how can we determine why it is so slow. We regularly have complaints that the developers need to wait for minutes (literally, not figuratively) for the diff results to show. The rest of the product seems fairly fast, but displaying a diff frequently takes more than 20 seconds, and on a regular basis even longer.

How can we work out what is taking the time in the Crucible stack? What sorts of things should be tuned?

We have 1GB allocated to the heap, and it is running on a 2GB machine, which according to the docs, provides performance "headroom". Our subversion repository is faster than this, but that shouldn't even be relevant given that Crucible has already indexed everything anyway?

What can we check and/or modify to improve this performance?

Accepted answers

All comments

Marek Parfianowicz

Hi Paul,

There may be many reasons, why an instance is slow. I would like to suggest checking the following things first:

1) Ensure that you have blame indexing enabled (the system property 'blame.calc.enable=true'; it's true by default). If not, then whenever FishEye has to display a diff with blame, it has to fetch it directly from SVN, which can be very time consuming. In case you have blame enabled, then it may happen that due to some indexing errors, blame data is not persisted for some files/revisions. In such case FishEye will also have to call SVN to fetch this data. There is an option to disable this fallback as well ('blame.scm.fallback.enable=false'). More details:

https://confluence.atlassian.com/fisheye/blame-calculations-785618348.html

2) Please check memory usage of your FishEye/Crucible instance. Check garbage collector logs. It's worth to ensure that 2GB of memory is indeed sufficient and that there are no long pauses caused by frequent full GCs. How many repositories do you have? What is their total size (GB)?

3) Please measure your network performance. For instance, try running a web browser directly on your FishEye server to see if overall performance is similar to one observed on workstations.

Cheers
Marek

Paul Wagland

We have blame indexing enabled. That said, it doesn't seem to work, we have been advised by Atlasssian to turn on enable the fallback, otherwise we have no blame calculation, which is not a workable solution.
I do not have access to the machine directly. Are the GC logs available through the user interface anywhere? According to our sys admins, most of the time the machine is idle, with occasional spikes of high CPU usage.
The network is not the issue. Most of the Crucible UI is perfectly functional, it only takes minutes to load the diff results, everything else is "fast", or at least "fast enough". For example, clicking on the admin page shows it in well under a second, opening the 'projects' overview shows the result in about a second on the first load. Going to "My Reviews" shows the page in under a second.

Marek Parfianowicz

ad 1. Please check if FishEye/Crucible logs contain any error messages related with repository indexing, such as:

"Problems calculating blame for <path>@&lt;rev>, will fetch from repository instead"

"Problems calculating blame for <path>@&lt;rev>, blame won't be calculated"

"Unable to annotate <path>@&lt;rev> as the content is not accessible"

"Problem committing author linecount, rolling back"

"Error reading database in linecount slurp"

Note:

enable debug logging first (in the administration panel)
it would be good to pick and reindex one repository (if this is not blocker for your team), to see how indexing was made for all commits (and not only new ones)

Counting amount of such errors will give you an indication how many commits are missing blame in FishEye's index.

Please also check how frequently the SCM fallback is used and how long it takes:

grep -E 'qtp.*<- getSvnBlameChunks.*' atlassian-*.log

Output will look like this (I wrapped the line for readability):

"DEBUG [qtp<thread id> ] fisheye Timer-output - <- getSvnBlameChunks(

rev=<rev>, contentKey=<path and rev>)

 chunks=110 time 144 millis <memory statistics>"

The "time XXX millis" says how long it took to ask SVN for blame.

Marek Parfianowicz

ad 2. There's no way to get it from UI. You'd have to add proper flags to a JVM and restart the server.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

fisheye-crucible

fisheye-crucible-administration