JIRA shuts down abruptly

Aitor Souto

Last week electrical power went away and JIRA server shut down. Restarting as usual did not work.

Logs say is having issues to connect to DB (MariaDB). I can connect remotely to the DB. And when I check connections, I can see Jira connected to DB.

I have tried to increase the connection pool, the validation query is there as well and timeout value in db variables is default: 28800

 

2018-05-21 16:33:03,654 localhost-startStop-1 ERROR [core.entity.jdbc.DatabaseUtil] Error getting column metadata for Error was:com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 3.150 milliseconds ago. The last packet sent successfully to the server was 3.150 milliseconds ago.. Not checking columns.
2018-05-21 16:33:03,654 localhost-startStop-1 ERROR [core.entity.jdbc.DatabaseUtil] Unable to close database connection, continuing anyway... Error was:java.sql.SQLException: Already closed.
2018-05-21 16:33:03,655 localhost-startStop-1 WARN [core.entity.jdbc.DatabaseUtil] Entity "Action" has 11 fields but table "jiraaction" has 0 columns.
2018-05-21 16:33:03,655 localhost-startStop-1 WARN [core.entity.jdbc.DatabaseUtil] Field "created" of entity "Action" is missing its corresponding column "CREATED"
2018-05-21 16:33:03,658 localhost-startStop-1 ERROR [NoModule] There was an error getting a DBCP datasource.
java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool
at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:59)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:243)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:74)
at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)
at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:146)
at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:136)
at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)
at org.ofbiz.core.entity.ConnectionFactory$1.getConnection(ConnectionFactory.java:52)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.getConnection(DatabaseUtil.java:132)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.addColumn(DatabaseUtil.java:1669)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.checkDb(DatabaseUtil.java:284)
at org.ofbiz.core.entity.jdbc.DatabaseUtil.checkDb(DatabaseUtil.java:144)
at org.ofbiz.core.entity.GenericDAO.checkDb(GenericDAO.java:1494)
at org.ofbiz.core.entity.GenericHelperDAO.checkDataSource(GenericHelperDAO.java:270)
at org.ofbiz.core.entity.GenericDelegator.initialiseAndCheckDatabase(GenericDelegator.java:236)
at org.ofbiz.core.entity.GenericDelegator.<init>(GenericDelegator.java:167)
at org.ofbiz.core.entity.GenericDelegator$1.load(GenericDelegator.java:91)
at org.ofbiz.core.entity.GenericDelegator$1.load(GenericDelegator.java:88)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3970)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)
at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)
at org.ofbiz.core.entity.GenericDelegator.getGenericDelegator(GenericDelegator.java:105)
at com.atlassian.core.ofbiz.CoreFactory.getGenericDelegator(CoreFactory.java:38)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.configureOfbiz(DatabaseConfigurationManagerImpl.java:238)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:342)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseConfigured(DatabaseConfigurationManagerImpl.java:221)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:115)
at com.atlassian.jira.startup.DefaultJiraLauncher.access$100(DefaultJiraLauncher.java:33)
at com.atlassian.jira.startup.DefaultJiraLauncher$1.run(DefaultJiraLauncher.java:86)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:34)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:81)
at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:72)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4992)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5490)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Andy Heinzer

I would recommend trying to follow this KB: https://confluence.atlassian.com/jirakb/mysql-communication-link-failure-214338032.html

Specifically cause #4.  I would be interested to make sure that first the interactive_timeout  and the wait_timeout values are for each of these SQL commands:

SHOW VARIABLES
SHOW SESSION VARIABLES
SHOW GLOBAL VARIABLES

If any of these 6 values are set to be an interval that is shorter than <time-between-eviction-runs-millis> in Jira's dbconfig.xml then this can explain these kinds of errors.   If that is the case, then what is happening is that the database server is dropping/terminating these connections prematurely and without Jira's awareness.  Jira is expecting to manage it's own database connection pool in terms of opening and allocating connections to the database.

Please let us know the results of those queries, and if you can I'd be interested to learn the content of your dbconfig.xml (you can remove the username/password and address if you are concerned about posting this to a public forum).

Could you also let us know what version of Jira this is, and what version of the JDBC mysql driver you are using?   The logs should tell both of these upon startup.