Hi everyone!
Since a couple of weeks now, we are experiencing that Jira v7.3.0#73011-sha1:3c73d0e hangs at around 1am and 1pm every day. Sometimes it is so bad we need to bring it down, erase plugins cache and re-start.
We have MariaDB 10.1.21 as the DB, with this config:
<?xml version="1.0" encoding="UTF-8"?>
<jira-database-config>
<name>defaultDS</name>
<delegator-name>default</delegator-name>
<database-type>mysql</database-type>
<jdbc-datasource>
<url>jdbc:mysql://localhost:3306/jira?useUnicode=true&characterEncoding=UTF8&sessionVariables=storage_engine=InnoDB</url>
<driver-class>com.mysql.jdbc.Driver</driver-class>
<username>XXX</username>
<password>XXX</password>
<pool-size>20</pool-size>
<pool-min-size>20</pool-min-size>
<pool-max-size>40</pool-max-size>
<pool-max-wait>30000</pool-max-wait>
<validation-query>select 1</validation-query>
<min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis>
<time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis>
<pool-max-idle>20</pool-max-idle>
<pool-remove-abandoned>true</pool-remove-abandoned>
<pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>
<pool-test-while-idle>true</pool-test-while-idle>
<pool-test-on-borrow>false</pool-test-on-borrow>
<validation-query-timeout>3</validation-query-timeout>
</jdbc-datasource>
</jira-database-config>
These are the some errors we see in the log file during these times:
2020-11-13 01:10:11,425 Caesium-1-4 ERROR anonymous Backup Service [c.a.j.bc.dataimport.DefaultExportService] Error during Active Objects Backup
com.atlassian.activeobjects.spi.ActiveObjectsImportExportException: There was an error during import/export with <unknown plugin>:
at com.atlassian.activeobjects.backup.ImportExportErrorServiceImpl.newImportExportSqlException(ImportExportErrorServiceImpl.java:26)
at com.atlassian.dbexporter.jdbc.JdbcUtils.withNoAutoCommit(JdbcUtils.java:57)
at
...
Caused by: java.sql.SQLException: Connection com.mysql.jdbc.JDBC4Connection@3fc49caa is closed.
at org.apache.commons.dbcp2.DelegatingConnection.checkOpen(DelegatingConnection.java:605)
at org.apache.commons.dbcp2.DelegatingConnection.setAutoCommit(DelegatingConnection.java:538)
And last but not least:
2020-11-13 13:09:09,262 commons-pool-EvictionTimer WARN [o.a.commons.dbcp2.BasicDataSource] An internal object pool swallowed an Exception.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 75,970 milliseconds ago. The last packet sent successfully to the server was 14 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:988)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3552)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3452)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3893)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:875)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1712)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1228)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2253)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:806)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
... 2 filtered
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:410)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:328)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureIdle(GenericObjectPool.java:920)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureMinIdle(GenericObjectPool.java:899)
at org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor.run(BaseGenericObjectPool.java:1036)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)
BackupService is scheduled with this cron parameters:
Backup Service : com.atlassian.jira.service.services.export.ExportService
Service Schedule : 0 0 1/12 * * ?
Thanks a lot for any tip!
Lior.
