Confluence server (6.8.1) shut down and restarted

[Rahul Pudi]

Confluence was down for sometime and got restarted . I am troubleshooting the reason for downtime. Could you please help me to understand the reason for the downtime. Please find the error log below:

--------------------------
Attributes
--------------------------
javax.servlet.forward.request_uri: /s/en_GB/7701/445126ab56584205167f1c035c86df495c7c4192/_/images/icons/macrobrowser/dropdown/html.png
javax.servlet.forward.context_path:
javax.servlet.forward.servlet_path: /s
javax.servlet.forward.path_info: /en_GB/7701/445126ab56584205167f1c035c86df495c7c4192/_/images/icons/macrobrowser/dropdown/html.png
javax.servlet.error.status_code: 500
com.atlassian.confluence.impl.profiling.DecoratorTimings: com.atlassian.confluence.impl.profiling.DecoratorTimings@3fb716c5
com.opensymphony.sitemesh.APPLIED_ONCE: true
__cleanup_recursion_counter: 0
javax.servlet.error.message:
javax.servlet.error.servlet_name: noop
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter_already_filtered: true
atlassian.core.seraph.original.url: /500page.jsp
com.atlassian.gzipfilter.GzipFilter_already_filtered: true
Confluence-Request-Time: 1557168971793
B3-TraceId: 62705059eb2ead
javax.servlet.error.request_uri: /s/en_GB/7701/445126ab56584205167f1c035c86df495c7c4192/_/images/icons/macrobrowser/dropdown/html.png
loginfilter.already.filtered: true
com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true
javax.servlet.error.exception: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
os_securityfilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true
--------------------------
Parameters
--------------------------
caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.atlassian.confluence.cache.hazelcast.hibernate.HazelcastHibernateCacheProvider.hazelcast(HazelcastHibernateCacheProvider.java:60)

2019-05-06 14:56:11,796 WARN [http-nio-8090-exec-41] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
-- traceId: f9c0267f44b34852
2019-05-06 14:56:11,796 WARN [http-nio-8090-exec-48] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
-- traceId: 7324e8b5698b43a6
2019-05-06 14:56:12,216 WARN [http-nio-8090-exec-78] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- url: /errors.jsp | traceId: 6b022d4e3a99943d
2019-05-06 14:56:12,216 ERROR [http-nio-8090-exec-78] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- url: /errors.jsp | traceId: 6b022d4e3a99943d
2019-05-06 14:56:12,366 WARN [http-nio-8090-exec-11] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/errors.jsp | url: /favicon.ico | traceId: 358a09f20cd7a72b
2019-05-06 14:56:12,366 ERROR [http-nio-8090-exec-11] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/errors.jsp | url: /favicon.ico | traceId: 358a09f20cd7a72b
2019-05-06 14:56:14,708 WARN [http-nio-8090-exec-31] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/editpage.action?pageId=67736893 | url: /rest/api/content/67736893 | traceId: 220fb5e84b73b668
2019-05-06 14:56:14,709 ERROR [http-nio-8090-exec-31] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/editpage.action?pageId=67736893 | url: /rest/api/content/67736893 | traceId: 220fb5e84b73b668
2019-05-06 14:56:14,727 WARN [http-nio-8090-exec-81] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/createpage-entervariables.action?templateId=65372178&spaceKey=TM&title=&newSpaceKey=TM&fromPageId=65573069 | url: /rest/analytics/1.0/publish/bulk | traceId: 97552940f8c69469
2019-05-06 14:56:14,728 ERROR [http-nio-8090-exec-81] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/createpage-entervariables.action?templateId=65372178&spaceKey=TM&title=&newSpaceKey=TM&fromPageId=65573069 | url: /rest/analytics/1.0/publish/bulk | traceId: 97552940f8c69469
2019-05-06 14:56:14,751 WARN [http-nio-8090-exec-37] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/display/TPP/2019-05-07 | url: /errors.jsp | traceId: 78ddf55d52d0744a
2019-05-06 14:56:14,752 ERROR [http-nio-8090-exec-37] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/display/TPP/2019-05-07 | url: /errors.jsp | traceId: 78ddf55d52d0744a
2019-05-06 14:56:14,762 WARN [http-nio-8090-exec-70] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/display/TPP/2019-05-07 | url: /errors.jsp | traceId: 3278a1b2a2bd8147
2019-05-06 14:56:14,763 ERROR [http-nio-8090-exec-70] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/display/TPP/2019-05-07 | url: /errors.jsp | traceId: 3278a1b2a2bd8147
2019-05-06 14:56:14,775 WARN [http-nio-8090-exec-1] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/editpage.action?pageId=67736893 | url: /errors.jsp | traceId: 837a90d02cd7f970
2019-05-06 14:56:14,776 ERROR [http-nio-8090-exec-1] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/editpage.action?pageId=67736893 | url: /errors.jsp | traceId: 837a90d02cd7f970
2019-05-06 14:56:14,795 WARN [http-nio-8090-exec-38] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/editpage.action?pageId=67736893 | url: /errors.jsp | traceId: 7066b19f3a74bf9d
2019-05-06 14:56:14,795 ERROR [http-nio-8090-exec-38] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/editpage.action?pageId=67736893 | url: /errors.jsp | traceId: 7066b19f3a74bf9d
2019-05-06 14:56:15,543 WARN [http-nio-8090-exec-91] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/createpage.action?spaceKey=ES&fromPageId=63475460 | url: /rest/analytics/1.0/publish/bulk | traceId: 3e7780eb0ec2989e
2019-05-06 14:56:15,543 ERROR [http-nio-8090-exec-91] [confluence.impl.vcache.SynchronousExternalCache] lambda$put$4 Failed to put entry into cache 'com.atlassian.confluence.locale.requestLang': Failed due to UNCLASSIFIED_FAILURE
-- referer: https://confluence.pncint.net/pages/createpage.action?spaceKey=ES&fromPageId=63475460 | url: /rest/analytics/1.0/publish/bulk | traceId: 3e7780eb0ec2989e
2019-05-06 14:56:15,546 INFO [http-nio-8090-exec-91] [atlassian.confluence.status.SystemErrorInformationLogger] writeToLog
Request Unique ID : fb244777-773a-4dbe-b8e0-881469443bfc

caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473)
caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.atlassian.confluence.cache.hazelcast.hibernate.HazelcastHibernateCacheProvider.hazelcast(HazelcastHibernateCacheProvider.java:60)

2019-05-06 14:56:41,298 WARN [http-nio-8090-exec-91] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
-- traceId: 4d8f639cfec0b9f3
2019-05-06 14:56:42,594 WARN [localhost-startStop-4] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
2019-05-06 14:56:42,596 WARN [localhost-startStop-4] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
2019-05-06 14:56:42,597 WARN [localhost-startStop-4] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE

at com.atlassian.confluence.schedule.ScheduleUtil.pauseAndFlushSchedulerService(ScheduleUtil.java:122)
at com.atlassian.confluence.util.SchedulerLifecycle.shutdown(SchedulerLifecycle.java:50)
at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.shutDown(DefaultLifecycleManager.java:95)
at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.shutDown(DefaultLifecycleManager.java:60)
at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextDestroyed(LifecycleServletContextListener.java:23)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4908)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5545)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.atlassian.scheduler.SchedulerServiceException: The scheduler service has been shut down; it cannot be restarted.
at com.atlassian.scheduler.core.AbstractSchedulerService.standby(AbstractSchedulerService.java:172)
at com.atlassian.confluence.schedule.ScheduleUtil.pauseAndFlushSchedulerService(ScheduleUtil.java:120)
... 13 more
2019-05-06 14:56:42,832 WARN [localhost-startStop-4] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
2019-05-06 14:56:42,835 INFO [localhost-startStop-4] [atlassian.plugin.manager.DefaultPluginManager] shutdown Preparing to shut down the plugin system
2019-05-06 14:56:42,871 WARN [localhost-startStop-4] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
2019-05-06 14:56:42,871 INFO [localhost-startStop-4] [atlassian.plugin.manager.DefaultPluginManager] shutdown Shutting down the plugin system
2019-05-06 14:56:42,949 WARN [localhost-startStop-4] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE