nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

connection pool exceptions - Continuition

srinivasp

I posted a question couple of minutes back and am contining here since i am unable to access that post.

Hi Dieter,

Please find the threaddump attached. I am unable to attach the complete dump but i found nearly 150 threads blocked. (dump1.txt)

Accepted answers

All comments

JamieA

I don't think you're going to get a correct answer based on the information in your question.

But, if you install the java melody plugin, it will show you all the database connections, along with stacktraces from what created the connection. Using this, you can find out which plugin (it probably is a plugin) is creating them.

JamieA

With java melody you should see two thread dumps for every connection, one is from when the connection was created, and one is what it is currently doing. The one that created it is the interesting one.

srinivasp

Hi Jamie,

Please add your comments to this post. I am unable to open the actual post(https://answers.atlassian.com/questions/52523/connection-pool-exceptions) when i click the link on your comments email. This is another issue i posted to atlassian. Melody plugin is already installed and I am analysing it and so far no luck yet. I cant attach here because it is huge.

Hi EvaLee,

I am using Jira 4.4.5 with oracle database.

Thanks

srinivasp

i copied the threads from the pdf. Please find the attached. (dump2.txt)

srinivasp

Hi Jamie, I found this -XX:+HeapDumpOnOutOfMemoryError in the vm arguments information. Will it be the cause?(systeminfo.txt)

Radu Dumitriu

1) 150 threads and 300 connections ? This is wrong. N connections < N threads (one thread should use at most one connection).

2) From your partial thread dump, I see that the threads are blocked on the monitor

waiting to lock <0x000000059a5bf828> (a org.apache.commons.dbcp.PoolableConnectionFactory) on validate.

2.a) you need to look in your thread dump for the string: locked 0x000000059a5bf828 (same address as above). Please
 show us that thread, this is the one that is blocking everything
2.b) you need to look a bit over the configuration of the DBCP datasource. Are you using validation? testWhileIdle ? Would 
be helpful to show us the configuration, too
2.c) Disable any 3rd party plugins before doing these tests.

JamieA

That's a good point... I found using a validation query made things massively slower (on oracle at any rate). The validation query is run before every single normal query. As Radu says 300 connections seems way too much... I manage a jira site with 10k users and the connection pool is 30.

srinivasp

 HI Radu/Jamie, Please find the db configuration below.

&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;jira-database-config&gt;
&lt;name&gt;defaultDS&lt;/name&gt;
&lt;delegator-name&gt;default&lt;/delegator-name&gt;
&lt;database-type&gt;oracle10g&lt;/database-type&gt;
&lt;jdbc-datasource&gt;
&lt;url&gt;jdbc:oracle:thin:@xxxxxxxxxxxxxx:1571:JIRAPROD&lt;/url&gt;
&lt;driver-class&gt;net.bull.javamelody.JdbcDriver&lt;/driver-class&gt;
&lt;connection-properties&gt;driver=oracle.jdbc.OracleDriver&lt;/connection-properties&gt;
&lt;username&gt;jira44app&lt;/username&gt;
&lt;password&gt;jira44app&lt;/password&gt;
&lt;removeAbandoned&gt;true&lt;/removeAbandoned&gt;
&lt;removeAbandonedTimeout&gt;600&lt;/removeAbandonedTimeout&gt;
&lt;pool-size&gt;300&lt;/pool-size&gt;
&lt;connection-properties&gt;SetBigStringTryClob=true&lt;/connection-properties&gt;
&lt;validation-query&gt;select 1 from dual&lt;/validation-query&gt;
&lt;/jdbc-datasource&gt;
&lt;/jira-database-config&gt;

Radu Dumitriu

How about that thread dump ?

Radu Dumitriu

In the absence of the thread dump, here is a wild educated guess:

1) 300 connections are way too much. You may have messages on ORCL side that it is in a kind of limbo.

2) Remove abandoned has something to do with your deadlock (remove abandoned is called AFAIK from another thread that the one that aquired the connection). The ORCL drivers are sensitive to these kinds of manipulation.

Actions:

1) Delete these lines

<removeAbandoned>true</removeAbandoned>

<removeAbandonedTimeout>600</removeAbandonedTimeout>

2) Edit this line:

<pool-size>35</pool-size>

3) Restart Jira & retest

srinivasp

Hi Dumitru,

Thanks for your suggestion. As we are frequently getting connection timed out exceptions, atlassian support guys suggested this.

JamieA

I would also remove the validation query.

srinivasp

We migrated two weeks back and observed this issue. We had similar configuration except the removeAbandoned property and there was absolutely no issues with the previous jira version.