Communications link failure

[Vincent Raaijmakers]

Hi, we keep getting this error. We followed all of the recommendations provided by the log analyzer and made sure we confirm to the requirements.

MySql runs local on the same server, that should exclude network errors.

Database driver: mysql-connector-java-5.1.44

MySQL version: 5.7.21

In dbconfig.xml we have the validation query:

<validation-query>select 1</validation-query>

    <min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis>

    <time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis>

    <pool-max-idle>20</pool-max-idle>

    <pool-remove-abandoned>true</pool-remove-abandoned>

    <pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>

    <pool-test-while-idle>true</pool-test-while-idle>

    <validation-query-timeout>3</validation-query-timeout>

We also (re)index frequently.

Yet, the # timeout errors are increasing. What can we do more to avoid these errors?

Thanks!

Vincent

[Vincent Raaijmakers · Accepted solution]

Thanks Andrew, I'll increase the values and let you know how it all worked out.

Thanks!!

Vincent

[Andy Heinzer · Accepted solution]

What version of Jira is this?

Why are you indexing frequently? 

I would recommend that you setup your dbconfig.xml to look as closely as possible to the sample listed in Connecting Jira to MySQL.

This communication link failure error tends to happen because Jira is expecting to handle its own database connection pool and does this on the set intervals of those eviction parameters.  The problem you are likely seeing is that your MySQL variables called 'wait_timeout' and/or 'interactive_timeout' are set to be a time period that is shorter than the eviction process timing Jira is trying to do.

In essence, MySQL is configured to kill these idle connections before Jira can clean them up by its own processes.   By default MySQL is set to a value of 28800 (in seconds) which equals once every 8 hours where it does this kind of connection clean up. 

Jira is using milliseconds to do these evictions, so typically every 300000 ms = 5 minutes with these settings.  I'm guessing your MySQL has an interval set currently that is probably less than 10 minutes if you're seeing this sometimes, and probably 5 minutes or less if you're seeing this a lot.  The problem comes because Jira is not aware the MySQL has killed this connection.  So Jira still believes it has to cleanup and/or reuse this connection and it cannot at that point. Hence you see this error in the Jira logs. 

In short, I would recommend looking at the KB: Connection Timeout with MySQL Database and then adjusting the MySQL variables to a larger value than Jira's eviction timing.

Granted that is a KB for the Stash product using MySQL, but the concepts are the same in Jira.  I remember the tricky part of this is that you tend to have to look at all three sections in MySQL to find where the variable has been shortened.

Note that MySQL retains several scopes for the wait_timeout and interactive_timeout variable. Double check the same setting of the variable by running the following queries on the database:

1. SHOW VARIABLES
2. SHOW SESSION VARIABLES
3. SHOW GLOBAL VARIABLES

[Vincent Raaijmakers]

Thanks for your prompt response Andrew!

We are using Jira 7.5.0. We only perform indexing when JIra suggests we need to do so.

I have no exact time period, but on average once every 2 months or so? 

So our MySQL variables are:

Show variables & Show session variables:

wait_timeout : 600 

Show global variables:

wait_timeout : 180

What would you advise?

Vincent

[Vincent Raaijmakers]

FYI, I do not think we have stash configured at all. If we search for stash-properties, nothing came back. Is it required to have stash, or can we do without?

This is what we have:

locate stash

/opt/fecru-4.5.0/licenses/com.atlassian.stash--stash-java-client-api--1.5.0.txt

/opt/fecru-4.5.0/licenses/com.atlassian.stash--stash-java-client-applinks--1.5.0.txt

/opt/fecru-4.5.0/licenses/com.atlassian.stash--stash-java-client-core--1.5.0.txt

/opt/fecru-4.5.0/var/plugins/bundled/fecru-stash-integration-plugin-4.5.0-20170908094439.jar

/usr/lib/git-core/git-stash

/usr/share/man/man1/git-stash.1.gz

/usr/src/linux-aws-headers-4.4.0-1022/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1026/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1031/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1032/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1035/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1038/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1039/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1041/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1043/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1044/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1047/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1048/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1049/arch/powerpc/include/asm/fsl_pamu_stash.h

/usr/src/linux-aws-headers-4.4.0-1052/arch/powerpc/include/asm/fsl_pamu_stash.h

/var/fecru-4.5.0/licenses/com.atlassian.stash--stash-java-client-api--1.5.0.txt

/var/fecru-4.5.0/licenses/com.atlassian.stash--stash-java-client-applinks--1.5.0.txt

/var/fecru-4.5.0/licenses/com.atlassian.stash--stash-java-client-core--1.5.0.txt

/var/fecru-4.5.0/var/plugins/bundled/fecru-stash-integration-plugin-4.5.0-20170908094439.jar

So 

[Andy Heinzer]

I realize you do not have Stash.  That KB was just an example of how using a MySQL database that is configured in that way can cause these kinds of communication failure errors.  I would not recommend trying to follow the specific resolution in that KB that talks about the stash.properties file.

Instead I suggest that you increase both the wait_timeout and interactive_timeout values on the MySQL side so that they are at least double the eviction internal Jira is using.  So in this case, I would expect these to both be at least 600.

But you might need to change this in 3 different locations of MySQL.  Hence each SQL command

SHOW VARIABLES

SHOW SESSION VARIABLES

SHOW GLOBAL VARIABLES

will return a bunch of variables and their values.  It is possible you might have different values for both wait_timeout and interactive_timeout on each of these three locations.

It's probably a good idea to first stop Jira, then make these SQL changes, before you then start Jira up again.