Recurring Crash of Confluence

[Armin Hüneburg]

Our Confluence setup was stable during the first few weeks but maybe a month ago it started to crash. At first it was once per week but now it is every second day. According to the logs, there are some job runners missing but I don't know whether they can cause confluence to crash. These include (not necessarily complete):

• amqJobRunner
• 24ec482d-9c69-43bd-bcdf-13e619147a0c-CrowdDirectorySynchroniserJobRunner
• confluence.extra.webdav:confluenceDavSessionInvalidator
• confluence.extra.webdav:contentJobQueueExecutor
• com.atlassian.confluence.plugins.confluence-daily-summary-email:summaryEmail

Last 2,000 lines of log (includes restart of confluence): https://pastebin.com/ndzveErR

The crash occurred between 00:16:06,201 (last log entrance) and 9:50 (someone messaged me that confluence is not reachable)

Given the regularity of the log entrances I guess that 00:16:06,201 or some error in its vicinity was responsible for the crash.

[AnnWorley · Accepted solution]

Hi Armin,

Thanks for the log and detailed description.

Theory

My theory is that you may have migrated from Cloud, and the automatic backup job was migrated from Cloud but is not compatible with the server instance. The automatic backup was the last job to fail before the restart:

2017-09-19 00:16:06,201 ERROR [Caesium-1-1] [scheduler.caesium.impl.CaesiumSchedulerService] executeClusteredJobWithRecoveryGuard Unhandled exception during the attempt to execute job 'amq-ProgressAwareTaskService:com.atlassian.ondemand.backupmanager.longrunning.SiteExportTaskRunner9118ced6-123a-4d02-ac92-677382efe6c8'; will attempt recovery in 60 seconds

Recommendations

• To eliminate the "executeClusteredJobWithRecoveryGuard Unhandled exception" errors please see the workaround on There are many errors "executeClusteredJobWithRecoveryGuard Unhandled exception" in atlassian-confluence.log after migrating from Cloud to Server. The workaround mentions backing up the database first, I would like to emphasize that step, too.
• Going forward, I suggest that you disable the automatic XML backups and implement a production backup strategy as described in Production Backup Strategy

I look forward to hearing how it goes.

Thanks,

Ann

[Armin Hüneburg]

Thank you for your help.

It seems to have worked (although automatic backups were already disabled...). While the log was cluttered with many exceptions, none were logged during the past 20 minutes. I hope this DB cleanup helps...

Thanks again.

[AnnWorley]

I am glad Confluence appears to be stable now.

If there is another crash we can dig more deeply into the logs and configuration files, so please keep us posted.

[Armin Hüneburg]

Ok, thanks again for your help. It is stable now, there are no more log messages about missing job runners and it didn't crash.