HTTP Status 500 - Cannot call sendError() after the response has been committed

[Lessandro]

I've a confluence cluster env. with 2 nodes.

Everything works fine when I start only the first node, but when I start the second one, I'm getting this error:

HTTP Status 500 - com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!

I've already increased the JAVA_Memory.

FULL LOG:

type Exception report

message com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE

description The server encountered an internal error that prevented it from fulfilling this request.

exception

java.util.concurrent.CompletionException: com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)
com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
root cause

com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE
com.atlassian.vcache.internal.legacy.LegacyUtils.mapException(LegacyUtils.java:51)
com.atlassian.vcache.internal.legacy.LegacyStableReadExternalCache.mapException(LegacyStableReadExternalCache.java:125)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:93)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:69)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.put(AbstractStableReadExternalCache.java:239)
com.atlassian.vcache.internal.core.metrics.TimedExternalWriteOperationsUnbuffered.put(TimedExternalWriteOperationsUnbuffered.java:36)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
root cause

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:104)
com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:99)
com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:93)
com.hazelcast.spi.AbstractDistributedObject.toData(AbstractDistributedObject.java:67)
com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:126)
com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:117)
com.atlassian.cache.hazelcast.HazelcastCache.put(HazelcastCache.java:115)
com.atlassian.confluence.cache.DefaultConfluenceCache.put(DefaultConfluenceCache.java:58)
com.atlassian.confluence.cache.ConfluenceMonitoringCache.put(ConfluenceMonitoringCache.java:87)
com.atlassian.vcache.internal.legacy.LegacyUtils.directPut(LegacyUtils.java:38)
com.atlassian.vcache.internal.legacy.LegacyStableReadExternalCache.internalPut(LegacyStableReadExternalCache.java:64)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$null$17(AbstractStableReadExternalCache.java:242)
com.atlassian.vcache.internal.core.service.VCacheLock.withLock(VCacheLock.java:33)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$put$18(AbstractStableReadExternalCache.java:242)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:83)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:69)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.put(AbstractStableReadExternalCache.java:239)
com.atlassian.vcache.internal.core.metrics.TimedExternalWriteOperationsUnbuffered.put(TimedExternalWriteOperationsUnbuffered.java:36)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.50 logs.

[Shannon S]

Hello!

Could you tell us, is this a fresh install of Confluence Data Center, or did you originally have a Confluence on a single server deployment that you are trying to migrate to Data Center?

Regards,

Shannon

[Lessandro]

Hi Shannon, thanks for answer me.

Yes, this is a fresh install. What I'm doing is:

1- setup the first node;

2- start the first node;

3- setup all settings(cluster name, db, etc)

4-Shutdown it down;

5- Copy to the second node;

6- Start the first node;

7- Start the second node.

Obs.: Everything is being building in AWS.

[AnnWorley]

Hi Lessandro,

Please check on the host VM of the second node: can it ping to the IP of the first node and reach it? (If not, please ensure that both nodes are in the same security group and can "talk" to each other)

If you are using quickformation templates, please check your setup against the following guide: Running Confluence Data Center in AWS

Thanks,

Ann

[Lessandro]

Hi Ann, 

Yes, both can "talk" to each other.

I'm not using quickformation. 

[Bernabe Theodore Costales III]

Hi @Lessandro:

If this is your first time setting up Confluence Data Center in AWS we would recommend using the QuickFormation templates that @AnnWorley had linked. However, it looks like you're attempting to configure this manually.

Based on the errors you posted in your original comment, my guess is that both nodes are able to talk to the database but not each other on the common ports that Confluence uses. This can cause a Cluster Panic event, which forces nodes in the cluster to terminate if they observe database writes from a third party source.

How are you verifying that both nodes are "talking" to each other? Are you using ping, netstat, some other kind of configuration? In AWS, are both nodes in the same target group with traffic routing through an ELB? What is your cluster join type: unicast, multicast, or AWS IAM?

There are a lot of unknowns in this thread. If this is your first time installing Data Center, try using this guide and following it step by step:

Installing Confluence Data Center

If you are running into issues following these steps exactly, then let us know where in the steps you're having troubles so we can analyze further.

[Lessandro]

Thanks Ted!

I recreated the environment from scratch and now it is working. I believe my error was because I was setting the shared directory as home directory.

Anyway, thanks again.