Jira hangs every 12 hours at 1am and 1pm

[Lior Marantenboim]

Hi everyone!

Since a couple of weeks now, we are experiencing that Jira v7.3.0#73011-sha1:3c73d0e hangs at around 1am and 1pm every day. Sometimes it is so bad we need to bring it down, erase plugins cache and re-start.

We have MariaDB 10.1.21 as the DB, with this config:

<?xml version="1.0" encoding="UTF-8"?>

<jira-database-config>

  <name>defaultDS</name>

  <delegator-name>default</delegator-name>

  <database-type>mysql</database-type>

  <jdbc-datasource>

    <url>jdbc:mysql://localhost:3306/jira?useUnicode=true&amp;characterEncoding=UTF8&amp;sessionVariables=storage_engine=InnoDB</url>

    <driver-class>com.mysql.jdbc.Driver</driver-class>

    <username>XXX</username>

    <password>XXX</password>

    <pool-size>20</pool-size>

    <pool-min-size>20</pool-min-size>

    <pool-max-size>40</pool-max-size>

    <pool-max-wait>30000</pool-max-wait>

    <validation-query>select 1</validation-query>

    <min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis>

    <time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis>

    <pool-max-idle>20</pool-max-idle>

    <pool-remove-abandoned>true</pool-remove-abandoned>

    <pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>

    <pool-test-while-idle>true</pool-test-while-idle>

    <pool-test-on-borrow>false</pool-test-on-borrow>

    <validation-query-timeout>3</validation-query-timeout>

  </jdbc-datasource>

</jira-database-config>

These are the some errors we see in the log file during these times:

2020-11-13 01:10:11,425 Caesium-1-4 ERROR anonymous    Backup Service [c.a.j.bc.dataimport.DefaultExportService] Error during Active Objects Backup

com.atlassian.activeobjects.spi.ActiveObjectsImportExportException: There was an error during import/export with <unknown plugin>:

        at com.atlassian.activeobjects.backup.ImportExportErrorServiceImpl.newImportExportSqlException(ImportExportErrorServiceImpl.java:26)

        at com.atlassian.dbexporter.jdbc.JdbcUtils.withNoAutoCommit(JdbcUtils.java:57)

        at 


...


Caused by: java.sql.SQLException: Connection com.mysql.jdbc.JDBC4Connection@3fc49caa is closed.

        at org.apache.commons.dbcp2.DelegatingConnection.checkOpen(DelegatingConnection.java:605)

        at org.apache.commons.dbcp2.DelegatingConnection.setAutoCommit(DelegatingConnection.java:538)

 And last but not least:

2020-11-13 13:09:09,262 commons-pool-EvictionTimer WARN      [o.a.commons.dbcp2.BasicDataSource] An internal object pool swallowed an Exception.

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure




The last packet successfully received from the server was 75,970 milliseconds ago.  The last packet sent successfully to the server was 14 milliseconds ago.

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)

        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:988)

        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3552)

        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3452)

        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3893)

        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:875)

        at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1712)

        at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1228)

        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2253)

        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284)

        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083)

        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:806)

        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)

        ... 2 filtered

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)

        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:410)

        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:328)

        at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)

        at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)

        at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)

        at org.apache.commons.pool2.impl.GenericObjectPool.ensureIdle(GenericObjectPool.java:920)

        at org.apache.commons.pool2.impl.GenericObjectPool.ensureMinIdle(GenericObjectPool.java:899)

        at org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor.run(BaseGenericObjectPool.java:1036)

        at java.util.TimerThread.mainLoop(Timer.java:555)

        at java.util.TimerThread.run(Timer.java:505)

Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.

        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)

 BackupService is scheduled with this cron parameters:

         Backup Service                                : com.atlassian.jira.service.services.export.ExportService

              Service Schedule                              : 0 0 1/12 * * ?

 Thanks a lot for any tip!

Lior.

[Daniel Ebers · Accepted solution]

Am I getting it right you have a backup job running the same time?
In case this applies could you cross-check by disabling the backup temporarily?

If it turns out the backup job is the cause it might make sense looking deeper into it - for example using database own tools (mysqldump?) to do the backups.

In the meanwhile here is more documentation on this topic:
https://confluence.atlassian.com/adminjiraserver/backing-up-data-938847673.html

Cheers,
Daniel

[Lior Marantenboim]

Hi Daniel,

As it seems, the backup service was causing it, I confused the cron entry with that of Linux format "minute hour day" instead of "seconds minutes hour", so I didn't realise it was scheduled to run twice. We have moved the backup to run only once at 1am, which is outside of our primary working hours. It still appears to hang Jira.

I believe the best solution would be to just switch to our backup software for this. 

Thanks a lot for the help.

Bests

Lior.

[Reneesh Kottakkalathil]

Hi Lior,

Per Jira7.4 link, Maria DB is not supported https://confluence.atlassian.com/adminjiraserver074/supported-platforms-881683157.html

Are you aware of any changes that you think could have caused this issue?