HTTP Status 500 - Cannot call sendError() after the response has been committed

Question

I've a confluence cluster env. with 2 nodes.

Everything works fine when I start only the first node, but when I start the second one, I'm getting this error:

HTTP Status 500 - com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!

I've already increased the JAVA_Memory.

FULL LOG:

type Exception report

message com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE

description The server encountered an internal error that prevented it from fulfilling this request.

exception

java.util.concurrent.CompletionException: com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)
com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
root cause

com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE
com.atlassian.vcache.internal.legacy.LegacyUtils.mapException(LegacyUtils.java:51)
com.atlassian.vcache.internal.legacy.LegacyStableReadExternalCache.mapException(LegacyStableReadExternalCache.java:125)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:93)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:69)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.put(AbstractStableReadExternalCache.java:239)
com.atlassian.vcache.internal.core.metrics.TimedExternalWriteOperationsUnbuffered.put(TimedExternalWriteOperationsUnbuffered.java:36)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
root cause

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:104)
com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:99)
com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:93)
com.hazelcast.spi.AbstractDistributedObject.toData(AbstractDistributedObject.java:67)
com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:126)
com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:117)
com.atlassian.cache.hazelcast.HazelcastCache.put(HazelcastCache.java:115)
com.atlassian.confluence.cache.DefaultConfluenceCache.put(DefaultConfluenceCache.java:58)
com.atlassian.confluence.cache.ConfluenceMonitoringCache.put(ConfluenceMonitoringCache.java:87)
com.atlassian.vcache.internal.legacy.LegacyUtils.directPut(LegacyUtils.java:38)
com.atlassian.vcache.internal.legacy.LegacyStableReadExternalCache.internalPut(LegacyStableReadExternalCache.java:64)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$null$17(AbstractStableReadExternalCache.java:242)
com.atlassian.vcache.internal.core.service.VCacheLock.withLock(VCacheLock.java:33)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$put$18(AbstractStableReadExternalCache.java:242)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:83)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:69)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.put(AbstractStableReadExternalCache.java:239)
com.atlassian.vcache.internal.core.metrics.TimedExternalWriteOperationsUnbuffered.put(TimedExternalWriteOperationsUnbuffered.java:36)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.50 logs.

Lessandro · Answer

Thanks Ted!

I recreated the environment from scratch and now it is working. I believe my error was because I was setting the shared directory as home directory.

Anyway, thanks again.

Bernabe Theodore Costales III · Answer

Hi @Lessandro:

If this is your first time setting up Confluence Data Center in AWS we would recommend using the QuickFormation templates that @AnnWorley had linked. However, it looks like you're attempting to configure this manually.

Based on the errors you posted in your original comment, my guess is that both nodes are able to talk to the database but not each other on the common ports that Confluence uses. This can cause a Cluster Panic event, which forces nodes in the cluster to terminate if they observe database writes from a third party source.

How are you verifying that both nodes are "talking" to each other? Are you using ping, netstat, some other kind of configuration? In AWS, are both nodes in the same target group with traffic routing through an ELB? What is your cluster join type: unicast, multicast, or AWS IAM?

There are a lot of unknowns in this thread. If this is your first time installing Data Center, try using this guide and following it step by step:

Installing Confluence Data Center

If you are running into issues following these steps exactly, then let us know where in the steps you're having troubles so we can analyze further.

Lessandro · Answer

Hi Ann,

Yes, both can "talk" to each other.

I'm not using quickformation.