Why does one node have a constantly increasing pool of active database connections

[Peter Wells]

JIRA version: 8.9.0

1763 users

Database SQL Server 2016

Our DataCenter installation of JIRA has 3 nodes running behind a load balancer. Each appear to have an equal share of the load, but while two nodes have low numbers of active connections, one has a high number of active connections which has a distincly upward trend:

All of the nodes have identical dbconfig.xml files:
<jira-database-config>
<name>defaultDS</name>
<delegator-name>default</delegator-name>
<database-type>mssql</database-type>
<schema-name>dbo</schema-name>
<jdbc-datasource>
<url>jdbc:sqlserver://;serverName=XXXXXXXXXX;portNumber=1433;databaseName=EPS-JIRA</url>
<driver-class>com.microsoft.sqlserver.jdbc.SQLServerDriver</driver-class>
<username>xxxxxx</username>
<password>xxxxxx</password>
<pool-min-size>10</pool-min-size>
<pool-max-size>60</pool-max-size>
<pool-max-wait>30000</pool-max-wait>
<validation-query>select 1</validation-query>
<min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis>
<time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis>
<pool-max-idle>20</pool-max-idle>
<pool-remove-abandoned>true</pool-remove-abandoned>
<pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>
<pool-test-on-borrow>false</pool-test-on-borrow>
<pool-test-while-idle>true</pool-test-while-idle>
</jdbc-datasource>
</jira-database-config>

I cannot find any differences in any other configuration between the nodes so I am at a loss as to why there is this marked difference. 

For reference, this is the monitoring on another node for the same time period:

Does anyone have an suggestions?

[mogavenasan]

Hi @Peter Wells,

The way it works in Jira Data Center is one of the nodes in the cluster will be performing those scheduled jobs, for example, the node could be running an indexing job.

However, the database connection should be released back into the pool once the job is completed successfully. 

Besides that, there could be some third-party plugin that running some of their own scheduled jobs which might be hogging the database connection pool.

Most of the time, this sort of scenario will need a Thread Dump analysis to identify what thread is using the database connection at the time the Thread Dump was generated.

I hope that this helps.

Thanks,
Moga