Weird JIRA HTTP Response Performance Issue

Question

I have JIRA 5.0.6 running with GreenHopper and Bonfire. I'm wrestling with an issue where HTTP responses for most write operations on most projects take about 20 seconds.

I know that it's the response that's hanging because I can immediately refresh the same issue in another window/tab and the issue is updated, whereas the original window/tab is still waiting for an HTTP response from the server.

Some projects aren't susceptible at all to this issue. And for the projects that are, adding people to an issue's watchlist is instantaneous.

I've already ruled out the database as a bottleneck; I initally thought the InnoDB buffer pool size was too small, but increasing it did nothing. The databse has since been migrated to a dedicated database server on a different machine, and it's using MS SQL instead of MySQL now. Connection pools are plentiful; nothing running low on resources. No performance improvement.

I've already ruled out memory as a bottleneck; I've increased max values on everything: heap size, PermGen space, etc. JVM memory graphs look fine.

I've already ruled out antivirus as a bottleneck; I completely disabled all antivirus scanning and nothing changed.

I enabled profiling briefly to determine what was happening. The log didn't show anything useful. Instead, there was just a substantial gap between one log entry and the next. (It's been a while, so I don't remember specifics anymore.)

Anybody else experience anything like this, or maybe have any ideas about what's going on here? Any chance this could be something event-related? (And if so, why aren't events handled asynchronously?)

Harry Chan5 · Answer

What is the CPU usage like? Have you checked the logs for any errors? Has all the memory been used up, e.g. by plugins?

This isn't right if it's slow even for 1 user. If you're using Java 6 and it's possible to switch to Java 7, this has made a huge difference for us.

Christopher Par9 · Answer

Migrating from MySQL (which was installed on the same box in an unsupported installation) to MS SQL (separate box, enterprise-level support from company's IT department) didn't affect performance one way or the other.

JIRA's installed on a 64-bit Windows Server 2008 machine.

The slowness is consistent, 24/7, and happens even if just one user is using the system (me). However, it happens for all users, regardless of browser (Chrome, Firefox, IE, Safari, Opera).

The JIRA instance is only accessible for internal users and there is no software firewall running on the box, so firewall/proxy wouldn't be affecting the connection.

I hadn't considered tuning Tomcat itself. It would make sense, considering Tomcat is what's controlling the HTTP lifecycle. I'll try that next. Thanks!

Harry Chan5 · Answer

Hi, there can be many causes to this still. Migrating to MS SQL instead of MySQL might not have helped. Have seen many more performance issues with JIRA on MS SQL due to how the locks work.

There are other paramaters of the JVM - have you looked at garbage collection, code cache size, if it's 32 or 64bit or other things?

Have you looked at tuning Tomcat? Are you running out of HTTP connections? Do you know how many users are online and using the system when you face the slowness? Is it consistent and 24/7? Do you have a reverse proxy sitting in front?

Have you enabled gzip compression and caching? As you have looked at JVM and database, I suggest you look at Tomcat, the web server, operating system etc.

It could also be firewall, network related etc.

You'd need to provide more information on the architecture / setup.

As it's HTTP response, it could also be the client. What browser are you using?

Performance tuning is a complicated excercise and if it all fails, it might be best to contact an expert on the matter.