nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

JIRA reads from storage massively - how to find what's causing?

Mariusz Kaczmarczyk

We are facing a serious performance degradation with our JIRA Software 7.6.17. Every once in a few days (or more recently every day) our JIRA application starts to generate a huge number of input/output (mostly read) operations, which increases with every minute and causes performance degradation severe enough to make us restart the system service. After restarting JIRA works with satisfying performance (kind of) - until another time.

We want to find out which JIRA component or operation causes this number of I/Os. Any idea how to do this? We use incoming mail extensively (44 accounts, some of them quite busy, ie. dozens of messages w/attachments per minute).

Unfortunately in our setup all of ${catalina.home}, ${jira.home} and ${jira.home}/caches reside on a single filesystem, so we cannot exactly tell where all these I/Os go to from virtual disks stats. We consider splitting the dirs, but it takes time and requires maintenance window. And maybe there's another, simpler solution, like increasing log level of certain JIRA components. Any help will be appreciated.

JIRA Software 7.6.17 and ServiceDesk 3.9.17 installed in a single instance.
OS: CentOS 7 x86_64, Java: Oracle JDK 1.8.0_102-b14
virtualization: ESXi 6.7, disk storage on a medium-class array with 10k HDDs, RAID5

UPDATE: turns out that with the latest incarnations of the issue we have lots more reads that writes. This rather eliminates logging from root causes.

Accepted answers

All comments

Aron Gombas _Midori_

Have you identified, at least, what files (if "storage" means the filesystem in this context) are written or read extensively?

Mariusz Kaczmarczyk

Yes, storage stands for JIRA application filesystem storage here. Not the database.

This is the essence of my question - how to identify directories/files that get this number of read/write operations. If we find the files, we're close to getting to the root cause.

Because of the scale (millions of files in ${jira.home} directory) any attempt to ex. find /jira/home/ -type f -mmin -1 -print takes hours. I can only think of tracing that using application logs, but I don't want to increase everything to DEBUG level, because doing so can make things even worse, ie. beat performance further because of increased log write rate.

Aron Gombas _Midori_

How about taking a thread dump in a moment when things are massive?

Maybe looking at the active threads could reveal what's going on? (And it could be cheaper or at least "expensive for a short period of time" than increasing the log level or watching the file system on the OS level...)

Mariusz Kaczmarczyk

OK, I could use Atlassian's support-data.sh tool and proceed with the dump. The script runs with errors, but thread dump seems to be working.

Not sure if I can analyze the dump - I'm definitely not an expert with Java. But I'll try.

Today we're mounting some throughput-sensitive directories (${jira.home}/caches/indexes most importantly) on a separate volume to measure i/o rate. Hope this sheds some light on the mystery.

Nic Brough -Adaptavist-

In my experience, I've seen this happen in three broad cases. Obviously, this is limited to my experience, so it could well be something else I have not encountered.

1. Swap

When a java process uses up too much RAM, it will cause the OS to make heavier use of "swap" space - using disk for things it can't keep in RAM.

If your people are triggering processes that outstrip the memory available, then you will see disk io massively increase while the Java process needs the extra space. The io should however drop back to normal when the process has finished.

This will also happen if you have something with a "memory leak". With a leak though, the disk access will not reduce. A memory leak means something is being allocated memory to work with, but it never releases that memory for other things to use. As more and more memory gets blocked out like that, your OS will increasingly use swap space, so you'll see io increasing until the system can't cope with it.

2. People

There are some functions in Jira that absolutely hammer your disks, notably the xml backup, indexing, and exports. Could your people be triggering actions that chew up the disk io?

3. Incorrect infrastructure

Someone put the index on a shared file system. Don't do that. It guarantees problems like the one you are seeing and can eventually corrupt your data.

I would look for those first, but even if you don't, my answer would not be to try to improve the disk performance, it would be to look at why Jira is doing this. That means looking at what files are being created/updated/destroyed during a period of high io.

Mariusz Kaczmarczyk

Thank you.

Swapping is not the issue, we have checked it already.

XML (application-based) backup and exports are not used at. Indexing is used and this is our primary suspect, so we want to move it to another filesystem to check how the I/Os split.

At the moment the whole ${jira.home} (together with caches/indexes) is on a single filesystem. This is a dedicated, per-VM, VMFS/VMDK-based virtual disk. No sharing involved (except for the shared disk array).

Mariusz Kaczmarczyk

The case is most likely solved. Here's what we've found.

we've moved caches/indexes from JIRA home to another mounted filesystem, to find out that it's Lucene indexing that's causing the massive reads, indeed.
the root cause was another app integrated with our JIRA instance that used to regularilly send hundreds of REST API requests like: /rest/api/2/search + no fields argument (no query string at all) in a few minutes.
our JIRA instance has over 800 dynamic (script) fields, implemented by Adaptivist ScriptRunner plugin, that were all generated and returned with every request (no fields parameter = ALL fields), with some of these fields even requiring another REST API call to the JIRA instance itself winding up the mess even further.

After we've disabled the other application (it's non-critical) the index filesystem read/write rate returned to acceptable values. We've reported the problem to their support and we're waiting for the fix. Our monitoring clearly shows that there should be no more issues like that in any near future.

As our JIRA instance runs behind a reverse proxy (Apache httpd-based) we're also implementing a guard to filter out all ^/rest/api/[0-9]+/search(/.+)$? requests without fields= query string arg., to enforce proper integration in future.

Aron Gombas _Midori_

Nice!

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

server

jira-software

jira-software-server