This is a simple enough question, but it probably has a complicated answer.
Given that we have a Crucible installation, running against subversion, how can we determine why it is so slow. We regularly have complaints that the developers need to wait for minutes (literally, not figuratively) for the diff results to show. The rest of the product seems fairly fast, but displaying a diff frequently takes more than 20 seconds, and on a regular basis even longer.
How can we work out what is taking the time in the Crucible stack? What sorts of things should be tuned?
We have 1GB allocated to the heap, and it is running on a 2GB machine, which according to the docs, provides performance "headroom". Our subversion repository is faster than this, but that shouldn't even be relevant given that Crucible has already indexed everything anyway?
What can we check and/or modify to improve this performance?
There may be many reasons, why an instance is slow. I would like to suggest checking the following things first:
1) Ensure that you have blame indexing enabled (the system property 'blame.calc.enable=true'; it's true by default). If not, then whenever FishEye has to display a diff with blame, it has to fetch it directly from SVN, which can be very time consuming. In case you have blame enabled, then it may happen that due to some indexing errors, blame data is not persisted for some files/revisions. In such case FishEye will also have to call SVN to fetch this data. There is an option to disable this fallback as well ('blame.scm.fallback.enable=false'). More details:
2) Please check memory usage of your FishEye/Crucible instance. Check garbage collector logs. It's worth to ensure that 2GB of memory is indeed sufficient and that there are no long pauses caused by frequent full GCs. How many repositories do you have? What is their total size (GB)?
3) Please measure your network performance. For instance, try running a web browser directly on your FishEye server to see if overall performance is similar to one observed on workstations.
ad 1. Please check if FishEye/Crucible logs contain any error messages related with repository indexing, such as:
"Problems calculating blame for <path>@<rev>, will fetch from repository instead"
"Problems calculating blame for <path>@<rev>, blame won't be calculated"
"Unable to annotate <path>@<rev> as the content is not accessible"
"Problem committing author linecount, rolling back"
"Error reading database in linecount slurp"
Counting amount of such errors will give you an indication how many commits are missing blame in FishEye's index.
Please also check how frequently the SCM fallback is used and how long it takes:
grep -E 'qtp.*<- getSvnBlameChunks.*' atlassian-*.log
Output will look like this (I wrapped the line for readability):
"DEBUG [qtp<thread id> ] fisheye Timer-output - <- getSvnBlameChunks(
rev=<rev>, contentKey=<path and rev>)
chunks=110 time 144 millis <memory statistics>"
The "time XXX millis" says how long it took to ask SVN for blame.
ad 1&3, as these are both going in the same direction.
None of the error messages mentioned at the top of the answer are found in our logs. Debugging has been enabled, but it won't be possible to re-index the repository until this evening. Our smallest repository can still be indexed overnight, so I will do that one and report back.
Looking at one of the files in one of the change sets that I know to be "slow". This file has 7009 lines, and is made up from 353 commits. Doing a 'time svn blame' on this file takes 3.5 seconds. Through the crucible UI this page takes over a minute to show.
This was done with debugging enabled, I'm waiting for our sysadmin to grant access to the file, and will investigate it then to look at the time taken for the svn blame.
Our SVN server is serving the data from a repository that is on normal HDD, and is not co-located on the same virtual machine as Crucible, although it is on a machine in the same VMWare cluster.
You can define GC options in the FISHEYE_OPTS system variable.
Thank you for the follow up information. At this point it is clear that the problem is that Crucible is not indexing our revisions for some reason. Because of this, the fallback is enabled, and that in turn is excruciatingly slow. This issue is being followed up with by a support issue.