nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

JIRA Data Center, connection pool of all nodes spike simultaneously, results in JIRA hangs

Ali Narenji

Dear Community,

We are using JIRA Data Center 7.9.2. Recently, the number of active database connections of all nodes spikes (~50) simultaneously in such a way that no nodes response any more. Number of active connections does not decrease until we stop the JIRA service of one of the nodes.

This incident occurs almost once a day.

No error is appeared in the JIRA log files during the incident, and no networking issue exists.

We have three nodes with Oracle 12c.

Thanks,

Ali

Accepted answers

Ali Narenji

Finally, we found the problem.

The newer versions of In-App & Desktop Notifications for Jira plugin caused the above issue to happen. After removing this plugin, hopefully everything gets back to normal.

As an alternative to the mentioned plugin, we installed Inbox: in-app notifications for Jira plugin which works much better than the previous one.

Many thanks for your hints,

Ali

All comments

JP _AC Bielefeld Leader_

Hi,

I would check the Jira services first:

https://confluence.atlassian.com/adminjiraserver073/services-867028234.html

Best

Ali Narenji

@JP _AC Bielefeld Leader_

Thanks for considering the issue.

No service is added or modified in Service page.

Mr J

Hi there Ali, is this happening consistently daily always at the same time? If yes, you may want to double-check the logs for anything that is happening in that time frame. As JP pointed out, there may be services running during that time that locked up your database connection pools. It might also be plugins, it can be hard to pinpoint as there are no errors but it may just be an abundance of cronjobs or something similar that is running too frequently at that timeframe, which would not trigger an error message.

You may also want to consider enabling DEBUG logging around that time to check what could be happening at that time. Please do remember to disable DEBUG logging after the issue has happened though as it can easily fill up your disk space and logs.

Ali Narenji

@Mr J

Dear J,

Thanks for giving the issue a great attention.

Unfortunately, the mentioned incident does not happen regularly. Sometimes it happens once a day in different hours. Sometimes it does not happen at all in a day. But we have realized that it does not happen when nobody uses JIRA (e.g., in off days).

Thanks,

Ali

Mr J

Hey there Ali, just to check, on off days does the Jira server gets shut down, or does it keep running? I wanted to confirm this because if the Jira server keeps running during off days and the issue doesn't happen, that would mean it's not a service or anything automated because those would run regardless of user activity.

In this case, it might be some sort of request or activity done by a user, making it a bit more tricky to track, especially as it doesn't happen at a regular time. With this new focus, is there anything in the logs that shows what activity was done before the nodes spike?

If there isn't, would you be alright with enabling DEBUG logs at least for a day (or at least until the incident happens)? I have to note that this is a very verbose logging level, so your logs will be filled up fast.

Ali Narenji

All nodes are always running even on off days.

Nothing special is shown in log files, and as you noted it is some how hard to enable the DEBUG level logs.

Just to clarify, I attached the connection pool graph of all three nodes. At 15:44, I stopped one of nodes, so the active connections of other nodes decreased significantly.

Ali Narenji

Finally, we found the problem.

The newer versions of In-App & Desktop Notifications for Jira plugin caused the above issue to happen. After removing this plugin, hopefully everything gets back to normal.