JIRA shuts down abruptly

[Aitor Souto]

Last week electrical power went away and JIRA server shut down. Restarting as usual did not work.

Logs say is having issues to connect to DB (MariaDB). I can connect remotely to the DB. And when I check connections, I can see Jira connected to DB.

I have tried to increase the connection pool, the validation query is there as well and timeout value in db variables is default: 28800

2018-05-21 16:33:03,654 localhost-startStop-1 ERROR [core.entity.jdbc.DatabaseUtil] Error getting column metadata for Error was:com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 3.150 milliseconds ago. The last packet sent successfully to the server was 3.150 milliseconds ago.. Not checking columns.
2018-05-21 16:33:03,654 localhost-startStop-1 ERROR [core.entity.jdbc.DatabaseUtil] Unable to close database connection, continuing anyway... Error was:java.sql.SQLException: Already closed.
2018-05-21 16:33:03,655 localhost-startStop-1 WARN [core.entity.jdbc.DatabaseUtil] Entity "Action" has 11 fields but table "jiraaction" has 0 columns.
2018-05-21 16:33:03,655 localhost-startStop-1 WARN [core.entity.jdbc.DatabaseUtil] Field "created" of entity "Action" is missing its corresponding column "CREATED"
2018-05-21 16:33:03,658 localhost-startStop-1 ERROR [NoModule] There was an error getting a DBCP datasource.
java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool
at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:59)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:243)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:74)
at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)
at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:146)
at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:136)
at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)
at org.ofbiz.core.entity.ConnectionFactory$1.getConnection(ConnectionFactory.java:52)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.getConnection(DatabaseUtil.java:132)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.addColumn(DatabaseUtil.java:1669)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.checkDb(DatabaseUtil.java:284)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.checkDb(DatabaseUtil.java:144)
at org.ofbiz.core.entity.GenericDAO.checkDb(GenericDAO.java:1494)
at org.ofbiz.core.entity.GenericHelperDAO.checkDataSource(GenericHelperDAO.java:270)
at org.ofbiz.core.entity.GenericDelegator.initialiseAndCheckDatabase(GenericDelegator.java:236)
at org.ofbiz.core.entity.GenericDelegator.<init>(GenericDelegator.java:167)
at org.ofbiz.core.entity.GenericDelegator$1.load(GenericDelegator.java:91)
at org.ofbiz.core.entity.GenericDelegator$1.load(GenericDelegator.java:88)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3970)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)
at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)
at org.ofbiz.core.entity.GenericDelegator.getGenericDelegator(GenericDelegator.java:105)
at com.atlassian.core.ofbiz.CoreFactory.getGenericDelegator(CoreFactory.java:38)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.configureOfbiz(DatabaseConfigurationManagerImpl.java:238)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:342)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseConfigured(DatabaseConfigurationManagerImpl.java:221)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:115)
at com.atlassian.jira.startup.DefaultJiraLauncher.access$100(DefaultJiraLauncher.java:33)
at com.atlassian.jira.startup.DefaultJiraLauncher$1.run(DefaultJiraLauncher.java:86)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:34)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:81)
at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:72)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4992)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5490)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

[Andy Heinzer]

I would recommend trying to follow this KB: https://confluence.atlassian.com/jirakb/mysql-communication-link-failure-214338032.html

Specifically cause #4.  I would be interested to make sure that first the interactive_timeout  and the wait_timeout values are for each of these SQL commands:

SHOW VARIABLES
SHOW SESSION VARIABLES
SHOW GLOBAL VARIABLES

If any of these 6 values are set to be an interval that is shorter than <time-between-eviction-runs-millis> in Jira's dbconfig.xml then this can explain these kinds of errors.   If that is the case, then what is happening is that the database server is dropping/terminating these connections prematurely and without Jira's awareness.  Jira is expecting to manage it's own database connection pool in terms of opening and allocating connections to the database.

Please let us know the results of those queries, and if you can I'd be interested to learn the content of your dbconfig.xml (you can remove the username/password and address if you are concerned about posting this to a public forum).

Could you also let us know what version of Jira this is, and what version of the JDBC mysql driver you are using?   The logs should tell both of these upon startup.

[Aitor Souto]

Hi Andrew, thanks for your reply, I have tried to modified dbconfig.xml to have same values, still is not working. 

The values of wait_timeout and interactive_timeout are in all cases 28800.

Driver version: 5.1.35 (Mysql connector)

DB version: MariaDB 5.5.56

Last week server was running with no issues. No changes have been applied into db or Jira configuration. I am suspicious there is something corrupted in the data but with the log I am checking cannot figure out what could be.

<?xml version="1.0" encoding="UTF-8"?>

<jira-database-config>
<name>defaultDS</name>
<delegator-name>default</delegator-name>
<database-type>mysql</database-type>
<jdbc-datasource>
<url>jdbc:mysql://localhost:3306/jira?useUnicode=true&amp;characterEncoding=UTF8&amp;sessionVariables=storage_engine=InnoDB</url>
<driver-class>com.mysql.jdbc.Driver</driver-class>
<username>USERNAME</username>
<password>PASSWORD</password>
<pool-min-size>20</pool-min-size>
<pool-max-size>20</pool-max-size>
<pool-max-wait>300000</pool-max-wait>
<validation-query>select 1</validation-query>
<min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis>
<time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis>
<pool-max-idle>20</pool-max-idle>
<pool-remove-abandoned>true</pool-remove-abandoned>
<pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>
<pool-test-while-idle>true</pool-test-while-idle>
<validation-query-timeout>3</validation-query-timeout>
</jdbc-datasource>
</jira-database-config>

[Andy Heinzer]

I don't see any problems with the configuration you have in the dbconfig.xml file for Jira.  I can see two different approaches to try to resolve this:

1. abandon the existing database, create a new database, and then import an XML backup of your Jira data into this new empty database
2. Alternatively, we can try to troubleshoot the existing setup further.   I would like to focus on the database at this point.  I found an interesting write up on this kind of problem and approaches to try to troubleshoot it at https://stackoverflow.com/questions/6865538/solving-a-communications-link-failure-with-jdbc-and-mysql

On a side note, MariaDB is not official supported by Atlassian per the Supported Platforms documentation.  So it might be better to migrate to a supported database, such as postgresql or mysql, this is one way to both try to resolve the problem and make sure you are using a supported platform.  Steps on how to do this are in Switching databases.   Personally, I would try #1 first, just because I would want to make sure that your platform is supported.  But it's up to you.  

[Aitor Souto]

Hi Andrew,

I have set up a new Jira instance and I have imported all the data from xml backup from last week. Now everything is working. I haven´t change database.

Thanks for your help.