Error when shutting down Confluence before backup

Andrew_Hall

Hi Everyone,

 

When shutting down Confluence prior to application backup I often receive the following error:

 

Using CATALINA_BASE:   /opt/atlassian/confluence 
Using CATALINA_HOME:   /opt/atlassian/confluence 
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp 
Using JRE_HOME:        /opt/atlassian/confluence/jre/ 
Using CLASSPATH:       /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar 
Using CATALINA_PID:    /opt/atlassian/confluence/work/catalina.pid 
Tomcat did not stop in time. 
To aid diagnostics a thread dump has been written to standard out. 
Killing Tomcat with the PID: 1359 
The Tomcat process has been killed. 

 I have encountered this issue before and had previously been advised by Atlassian support to modify /opt/atlassian/confluence/bin/stop-confluence.sh to change the default shutdown timeout (from 20 seconds to):

 

and

$sucmd -m $CONF_USER -c "$PRGDIR/shutdown.sh 120 -force $@"

 

This does not happen all the time but seems to be largely dependent on how long Confluence has been running. The above result was after 7 days of operation.

Additionally, I have been advised previously that it is best practice for the Confluence services to be stopped while taking a production backup ie stop confluence service, mysqldump, tar home and application paths, start confluence service. Are there any other options for doing this? Forcibly killing Tomcat all the time seems less than ideal and I'm a little afraid of data not being successfully flushed to disk by the time the stop-confluence.sh script forcibly kills Tomcat.

I have also tried setting a 180 second timeout in the shutdown script but the issue still occurs. Are there any additional diagnostic steps that can be done to figure out why Tomcat is not exiting cleanly? The server only receives very light traffic and there were zero people connected/editing at the time of the above shutdown.

 

Any advise you could provide would be much appreciated.

Regards

 

Andrew Hall

 

 

Daniel Eads

Hey Andrew,

Firstly with the shutdown command - this happens when the plugin subsystem is still running. It's really nothing to worry about. The main application has stopped serving requests at this point and we literally kill the pid so that the process doesn't just run in perpetuity waiting for non-responsive plugins to say they're dead. It's safe to not be concerned about that, and you can even set the kill timeout down lower if you want. We default to 20 seconds as you initially noted.

Now for backups! Mysqldump will take a hot backup with innodb (the database type we use for Confluence). You can also for the most part take hot backups of the home and install directories. So for daily backup purposes, you don't need to bring Confluence down to take a backup.

Restoring can get a little hairy though - there is  lock file that will be written to your backup (unless you take specific steps to omit it). When restoring and trying to bring Confluence up from a full backup, you'll need to delete the lock file before Confluence will start. It's at:

<confluence-home>/lock

Confluence will throw an error if you try and start it up and the lock file is still in place, so you will get some help if you forget to remove it. But you could also use flags in your tar command to omit it from your backup as well. In either case, no need to fully shut down the application to take a backup!

Cheers,
Daniel