how to diagnose why crucible is slow

[Paul Wagland]

This is a simple enough question, but it probably has a complicated answer.

Given that we have a Crucible installation, running against subversion, how can we determine why it is so slow. We regularly have complaints that the developers need to wait for minutes (literally, not figuratively) for the diff results to show. The rest of the product seems fairly fast, but displaying a diff frequently takes more than 20 seconds, and on a regular basis even longer.

How can we work out what is taking the time in the Crucible stack? What sorts of things should be tuned?

We have 1GB allocated to the heap, and it is running on a 2GB machine, which according to the docs, provides performance "headroom". Our subversion repository is faster than this, but that shouldn't even be relevant given that Crucible has already indexed everything anyway?

What can we check and/or modify to improve this performance?

[Marek Parfianowicz]

Hi Paul,

There may be many reasons, why an instance is slow. I would like to suggest checking the following things first:

1) Ensure that you have blame indexing enabled (the system property 'blame.calc.enable=true'; it's true by default). If not, then whenever FishEye has to display a diff with blame, it has to fetch it directly from SVN, which can be very time consuming. In case you have blame enabled, then it may happen that due to some indexing errors, blame data is not persisted for some files/revisions. In such case FishEye will also have to call SVN to fetch this data. There is an option to disable this fallback as well ('blame.scm.fallback.enable=false'). More details:

https://confluence.atlassian.com/fisheye/blame-calculations-785618348.html

2) Please check memory usage of your FishEye/Crucible instance. Check garbage collector logs. It's worth to ensure that 2GB of memory is indeed sufficient and that there are no long pauses caused by frequent full GCs. How many repositories do you have? What is their total size (GB)? 

3) Please measure your network performance. For instance, try running a web browser directly on your FishEye server to see if overall performance is similar to one observed on workstations.

Cheers
Marek 

[Paul Wagland]

1. We have blame indexing enabled. That said, it doesn't seem to work, we have been advised by Atlasssian to turn on enable the fallback, otherwise we have no blame calculation, which is not a workable solution.
2. I do not have access to the machine directly. Are the GC logs available through the user interface anywhere? According to our sys admins, most of the time the machine is idle, with occasional spikes of high CPU usage.
3. The network is not the issue. Most of the Crucible UI is perfectly functional, it only takes minutes to load the diff results, everything else is "fast", or at least "fast enough". For example, clicking on the admin page shows it in well under a second, opening the 'projects' overview shows the result in about a second on the first load. Going to "My Reviews" shows the page in under a second.

[Marek Parfianowicz]

ad 1. Please check if FishEye/Crucible logs contain any error messages related with repository indexing, such as:

"Problems calculating blame for <path>@<rev>, will fetch from repository instead"

"Problems calculating blame for <path>@<rev>, blame won't be calculated"

"Unable to annotate <path>@<rev> as the content is not accessible"

"Problem committing author linecount, rolling back"

"Error reading database in linecount slurp"

Note:

• enable debug logging first (in the administration panel)
• it would be good to pick and reindex one repository (if this is not blocker for your team), to see how indexing was made for all commits (and not only new ones)

Counting amount of such errors will give you an indication how many commits are missing blame in FishEye's index.

Please also check how frequently the SCM fallback is used and how long it takes:

grep -E 'qtp.*<- getSvnBlameChunks.*' atlassian-*.log

Output will look like this (I wrapped the line for readability):

"DEBUG [qtp<thread id> ] fisheye Timer-output - <- getSvnBlameChunks(

rev=<rev>, contentKey=<path and rev>)

 chunks=110 time 144 millis <memory statistics>"

The "time XXX millis" says how long it took to ask SVN for blame.

[Marek Parfianowicz]

ad 2. There's no way to get it from UI. You'd have to add proper flags to a JVM and restart the server. 

[Marek Parfianowicz]

ad 3. Judging from your description, fetching blame is the main bottleneck. 

Could you tell how your SVN is configured in FishEye? Does your SVN server run on the same machine as FishEye? What hard disk is used? It it SSD?

[Paul Wagland]

ad 1&3, as these are both going in the same direction.

None of the error messages mentioned at the top of the answer are found in our logs. Debugging has been enabled, but it won't be possible to re-index the repository until this evening. Our smallest repository can still be indexed overnight, so I will do that one and report back.

Looking at one of the files in one of the change sets that I know to be "slow". This file has 7009 lines, and is made up from 353 commits. Doing a 'time svn blame' on this file takes 3.5 seconds. Through the crucible UI this page takes over a minute to show.

This was done with debugging enabled, I'm waiting for our sysadmin to grant access to the file, and will investigate it then to look at the time taken for the svn blame.

Our SVN server is serving the data from a repository that is on normal HDD, and is not co-located on the same virtual machine as Crucible, although it is on a machine in the same VMWare cluster.

[Paul Wagland]

ad 2. Is there a page indicating where/what these changes should be?

[Marek Parfianowicz]

You can define GC options in the FISHEYE_OPTS system variable.

https://confluence.atlassian.com/display/FISHKB/Fix+Out+of+Memory+errors

https://confluence.atlassian.com/display/FISHEYE/Tuning+FishEye+performance

[Paul Wagland]

Thank you for the follow up information. At this point it is clear that the problem is that Crucible is not indexing our revisions for some reason. Because of this, the fallback is enabled, and that in turn is excruciatingly slow. This issue is being followed up with by a support issue.