HTTP Status 500 - Cannot call sendError() after the response has been committed

Lessandro March 22, 2018

I've a confluence cluster env. with 2 nodes.

Everything works fine when I start only the first node, but when I start the second one, I'm getting this error:

HTTP Status 500 - com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!

I've already increased the JAVA_Memory.

FULL LOG:

type Exception report

message com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE

description The server encountered an internal error that prevented it from fulfilling this request.

exception

java.util.concurrent.CompletionException: com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)
com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
root cause

com.atlassian.vcache.ExternalCacheException: Failed due to UNCLASSIFIED_FAILURE
com.atlassian.vcache.internal.legacy.LegacyUtils.mapException(LegacyUtils.java:51)
com.atlassian.vcache.internal.legacy.LegacyStableReadExternalCache.mapException(LegacyStableReadExternalCache.java:125)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:93)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:69)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.put(AbstractStableReadExternalCache.java:239)
com.atlassian.vcache.internal.core.metrics.TimedExternalWriteOperationsUnbuffered.put(TimedExternalWriteOperationsUnbuffered.java:36)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
root cause

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:104)
com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:99)
com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:93)
com.hazelcast.spi.AbstractDistributedObject.toData(AbstractDistributedObject.java:67)
com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:126)
com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:117)
com.atlassian.cache.hazelcast.HazelcastCache.put(HazelcastCache.java:115)
com.atlassian.confluence.cache.DefaultConfluenceCache.put(DefaultConfluenceCache.java:58)
com.atlassian.confluence.cache.ConfluenceMonitoringCache.put(ConfluenceMonitoringCache.java:87)
com.atlassian.vcache.internal.legacy.LegacyUtils.directPut(LegacyUtils.java:38)
com.atlassian.vcache.internal.legacy.LegacyStableReadExternalCache.internalPut(LegacyStableReadExternalCache.java:64)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$null$17(AbstractStableReadExternalCache.java:242)
com.atlassian.vcache.internal.core.service.VCacheLock.withLock(VCacheLock.java:33)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$put$18(AbstractStableReadExternalCache.java:242)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:83)
com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:69)
com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.put(AbstractStableReadExternalCache.java:239)
com.atlassian.vcache.internal.core.metrics.TimedExternalWriteOperationsUnbuffered.put(TimedExternalWriteOperationsUnbuffered.java:36)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:184)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:168)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:151)
com.atlassian.confluence.impl.vcache.SynchronousExternalCache.put(SynchronousExternalCache.java:138)
com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:213)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:45)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.50 logs.

1 answer

0 votes
Shannon S
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 23, 2018

Hello!

Could you tell us, is this a fresh install of Confluence Data Center, or did you originally have a Confluence on a single server deployment that you are trying to migrate to Data Center?

Regards,

Shannon

Lessandro March 23, 2018

Hi Shannon, thanks for answer me.

Yes, this is a fresh install. What I'm doing is:

1- setup the first node;

2- start the first node;

3- setup all settings(cluster name, db, etc)

4-Shutdown it down;

5- Copy to the second node;

6- Start the first node;

7- Start the second node.

Obs.: Everything is being building in AWS.

AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 26, 2018

Hi Lessandro,

 

Please check on the host VM of the second node: can it ping to the IP of the first node and reach it? (If not, please ensure that both nodes are in the same security group and can "talk" to each other)

If you are using quickformation templates, please check your setup against the following guide: Running Confluence Data Center in AWS

Thanks,

Ann

Lessandro March 26, 2018

Hi Ann, 

Yes, both can "talk" to each other.

I'm not using quickformation. 

Bernabe Theodore Costales III
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 26, 2018

Hi @Lessandro:

If this is your first time setting up Confluence Data Center in AWS we would recommend using the QuickFormation templates that @AnnWorley had linked. However, it looks like you're attempting to configure this manually.

Based on the errors you posted in your original comment, my guess is that both nodes are able to talk to the database but not each other on the common ports that Confluence uses. This can cause a Cluster Panic event, which forces nodes in the cluster to terminate if they observe database writes from a third party source.

How are you verifying that both nodes are "talking" to each other? Are you using ping, netstat, some other kind of configuration? In AWS, are both nodes in the same target group with traffic routing through an ELB? What is your cluster join type: unicast, multicast, or AWS IAM?

There are a lot of unknowns in this thread. If this is your first time installing Data Center, try using this guide and following it step by step:

Installing Confluence Data Center

If you are running into issues following these steps exactly, then let us know where in the steps you're having troubles so we can analyze further.

Lessandro April 4, 2018

Thanks Ted!

I recreated the environment from scratch and now it is working. I believe my error was because I was setting the shared directory as home directory.

Anyway, thanks again. 

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events