Confluence eats network resources

[jimmo42]

openSUSE 42.2
mysql 5.6.36
Confluence 6.0.6
Jira 7.3.1
BitBucket 4.14.1

Within 30 minutes after a reboot, I can no longer open new network connections. My CMS is not accessible, I cannot connect via SSH, and my external monitors give timeouts for HTTP and SMTP. (ping works)

There are sometimes over 100 connections between confluence and MySQL. (netstat -an | grep -v ':80'| awk '{print $5}' | grep 127.0.0.1:3306 | wc -l)
MySQL has 40 processes just for Confluence. (show processlist)

If I shutdown confluence and wait until the existing network connections disappear, I can log into my CMS and the external monitors stop reporting timeouts. When I start confluence again, I soon have the same problem. If I am logged into Confluence before the problem starts, I will eventually get "com.atlassian.crowd.exception.runtime.CommunicationException".

There is no one else on the system and only a couple of hits per *minute* on the CMS. In fact, there is only a single Confluence user: me! It makes no sense to me that Confluence uses that many network connections and that many MySQL processes. It does be doing close to nothing at the moment. It is not as if there is some scheduled action that regularly slows the system, but rather a continuous problem.

All of the database and network parameters are still the defaults. I have looked through the Confluence Admin pages (when they are accessible long enough) and I did not find anything that limits either network connections or database connections.

Any ideas as to why this problem is occurring and what I can do to correct it are greatly appreaciated.

[Vickey Palzor Lepcha]

I'd start by looking at server memory allocations - confluence memory and collaborative editing memory allocation to begin with :-) . 

[AnnWorley]

Also, I am curious about the database connection pool compared with allowed http threads - the materials I asked James for in the support ticket will allow me to see that and more. Afterward we can circle back to this thread.

[jimmo42]

Thank you for your reply. I cannot even login at this point.  I looked through the files in confluence/conf, but I didn't find anything that seems like something I should change.

[Vickey Palzor Lepcha]

Logs perhaps James - they speak volumes.

[jimmo42]

Hmmmm.....

14-Mar-2018 17:06:24.372 WARNING [localhost-startStop-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named
[C3P0PooledConnectionPoolManager[identityToken->mh3401r8ci70mj1fmw|4afebb42]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.util.TimerThread.mainLoop(Timer.java:552)
 java.util.TimerThread.run(Timer.java:505)

There are quite a few entries like this in catalina.out with different thread names. (Do you need the entire log?)

[AnnWorley]

Hi James,

I would like to examine logs and configuration files from your instance so I opened a support ticket. The email address you use for this forum should get an email from our support portal soon. After we get it figured out we can let the Community know our findings.

Thanks,

Ann