It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

JIRA Data Center, connection pool of all nodes spike simultaneously, results in JIRA hangs

Dear Community,

 

We are using JIRA Data Center 7.9.2. Recently, the number of active database connections of all nodes spikes (~50) simultaneously in such a way that no nodes response any more. Number of active connections does not decrease until we stop the JIRA service of one of the nodes.

This incident occurs almost once a day.

No error is appeared in the JIRA log files during the incident, and no networking issue exists.

We have three nodes with Oracle 12c.

 

Thanks,

Ali

3 answers

1 accepted

0 votes
Answer accepted

Finally, we found the problem.

The newer versions of  In-App & Desktop Notifications for Jira plugin caused the above issue to happen. After removing this plugin, hopefully everything gets back to normal.

As an alternative to the mentioned plugin, we installed Inbox: in-app notifications for Jira plugin which works much better than the previous one.

Many thanks for your hints,

Ali

Hi Ali,

The issue of In-App & Desktop Notifications for Jira has been solved since you have raised it. Thank you!

You can retry it again noting that you will get more features than the alternative one.

Best Regards,

Nourhan

@JP -AC Bielefeld Leader- 

Thanks for considering the issue.

No service is added or modified in Service page.

Hi there Ali, is this happening consistently daily always at the same time? If yes, you may want to double-check the logs for anything that is happening in that time frame.  As JP pointed out, there may be services running during that time that locked up your database connection pools. It might also be plugins, it can be hard to pinpoint as there are no errors but it may just be an abundance of cronjobs or something similar that is running too frequently at that timeframe, which would not trigger an error message.

You may also want to consider enabling DEBUG logging around that time to check what could be happening at that time. Please do remember to disable DEBUG logging after the issue has happened though as it can easily fill up your disk space and logs. 

@Mr J 

Dear J,

Thanks for giving the issue a great attention.

Unfortunately, the mentioned incident does not happen regularly. Sometimes it happens once a day in different hours. Sometimes it does not happen at all in a day. But we have realized that it does not happen when nobody uses JIRA (e.g., in off days).

Thanks,

Ali

Hey there Ali, just to check, on off days does the Jira server gets shut down, or does it keep running? I wanted to confirm this because if the Jira server keeps running during off days and the issue doesn't happen, that would mean it's not a service or anything automated because those would run regardless of user activity.

In this case, it might be some sort of request or activity done by a user, making it a bit more tricky to track, especially as it doesn't happen at a regular time. With this new focus, is there anything in the logs that shows what activity was done before the nodes spike?

If there isn't, would you be alright with enabling DEBUG logs at least for a day (or at least until the incident happens)? I have to note that this is a very verbose logging level, so your logs will be filled up fast. 

All nodes are always running even on off days.

Nothing special is shown in log files, and as you noted it is some how hard to enable the DEBUG level logs.

Just to clarify, I attached the connection pool graph of all three nodes. At 15:44, I stopped one of nodes, so the active connections of other nodes decreased significantly.

ConnectionPools-spikes.png

Hi Ali,

did you get in touch with Atlassian on this issue. I would open a support ticket with them, as this seems pretty special considering the absence of any errors in the logs.

Best

JP

Suggest an answer

Log in or Sign up to answer
Community showcase
Posted in Jira

Demo Den Ep. 7: New Jira Cloud Reports

Learn how to use two new reports for next-gen projects in Jira Cloud:  Cumulative flow diagram and Sprint burndown chart. Ivan Teong, Product Manager, Jira Software, demos the Cumulative ...

308 views 1 3
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you