HTTP 500 Internal Server Error - Confluence DC Upgrade with Vacant exception

[Maansi Chandira]

After upgrading Confluence DC from 7.7.2 to 7.8.1 - I get HTTP 500 internal server error.

Java version - 1.8.0_265

-----Logs------

Confluence:

com.atlassian.cache.CacheException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.

com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:133)

com.atlassian.confluence.cache.ehcache.DefaultConfluenceEhCache.get(DefaultConfluenceEhCache.java:47)

com.atlassian.confluence.cache.ConfluenceMonitoringCache.get(ConfluenceMonitoringCache.java:57)

com.atlassian.confluence.cache.TransactionalCacheFactory$TransactionalCache.get(TransactionalCacheFactory.java:343)

com.atlassian.confluence.impl.cache.tx.TransactionAwareCache$1.get(TransactionAwareCache.java:43)

com.atlassian.confluence.user.crowd.DefaultApplicationCache.getApplication(DefaultApplicationCache.java:77)

com.atlassian.confluence.user.crowd.CachedCrowdApplicationDao.findByName(CachedCrowdApplicationDao.java:41)

sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)

org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:16)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)

org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)

com.sun.proxy.$Proxy99.findByName(Unknown Source)

com.atlassian.crowd.embedded.core.CrowdEmbeddedApplicationFactory.getApplication(CrowdEmbeddedApplicationFactory.java:22)

com.atlassian.crowd.embedded.core.CrowdServiceImpl.getApplication(CrowdServiceImpl.java:649)

com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:65)

sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)

org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:16)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)

org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)

com.sun.proxy.$Proxy106.authenticate(Unknown Source)

com.atlassian.crowd.embedded.atlassianuser.EmbeddedCrowdAuthenticator.authenticate(EmbeddedCrowdAuthenticator.java:34)

bucket.user.DefaultUserAccessor.authenticate(DefaultUserAccessor.java:465)

sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)

org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:16)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)

org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)

com.sun.proxy.$Proxy86.authenticate(Unknown Source)

com.atlassian.confluence.user.ConfluenceAuthenticator.authenticate(ConfluenceAuthenticator.java:73)

com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:90)

com.atlassian.confluence.user.ConfluenceAuthenticator.login(ConfluenceAuthenticator.java:54)

com.atlassian.seraph.auth.DefaultAuthenticator.getUserFromBasicAuthentication(DefaultAuthenticator.java:530)

com.atlassian.seraph.auth.DefaultAuthenticator.getUser(DefaultAuthenticator.java:341)

com.atlassian.confluence.user.ConfluenceCrowdSSOAuthenticator.lambda$getUser$2(ConfluenceCrowdSSOAuthenticator.java:94)

io.atlassian.fugue.Suppliers$MemoizingSupplier.get(Suppliers.java:175)

com.atlassian.confluence.user.ConfluenceCrowdSSOAuthenticator.getUser(ConfluenceCrowdSSOAuthenticator.java:103)

com.atlassian.seraph.auth.AbstractAuthenticator.getUser(AbstractAuthenticator.java:45)

com.atlassian.seraph.filter.BaseLoginFilter$SecurityHttpRequestWrapper.getUserPrincipal(BaseLoginFilter.java:238)

com.atlassian.seraph.filter.BaseLoginFilter$SecurityHttpRequestWrapper.getRemoteUser(BaseLoginFilter.java:225)

com.atlassian.confluence.util.UserNameHeaderFilter.doFilter(UserNameHeaderFilter.java:20)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.web.filter.MauEventFilter.doFilterInternal(MauEventFilter.java:34)

com.atlassian.confluence.web.filter.AbstractStaticResourceAwareFilter.doFilter(AbstractStaticResourceAwareFilter.java:43)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.web.filter.ConfluenceTimeoutFilter.doFilter(ConfluenceTimeoutFilter.java:57)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.web.filter.HttpSessionRegistrarFilter.doFilter(HttpSessionRegistrarFilter.java:44)

com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)

com.atlassian.confluence.web.filter.ThreadLocalCacheFilter.doFilter(ThreadLocalCacheFilter.java:25)

com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter$SwapOnBootstrapFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:55)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:35)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)

com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)

com.atlassian.confluence.util.ClusterHeaderFilter.doFilter(ClusterHeaderFilter.java:57)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.web.filter.ConfluenceOpenSessionInViewFilter.doFilterInternal(ConfluenceOpenSessionInViewFilter.java:38)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

com.atlassian.util.profiling.filters.ProfilingFilter.doFilter(ProfilingFilter.java:105)

com.atlassian.confluence.impl.servlet.RequestEventPublishingFilter.doFilter(RequestEventPublishingFilter.java:41)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)

com.atlassian.core.filters.cache.AbstractCachingFilter.doFilter(AbstractCachingFilter.java:31)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)

com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)

com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)

com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:67)

com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)

com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)

com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)

com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)

com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)

com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)

[Daniel Ebers]

Hi Maansi,

from some other Community posts the solution was to re-check the file system permission of the Confluence home directory. As you said you faced the error during upgrade maybe this was just forgotten:

• Confluence installation throw an error : "Confluence is vacant, a call to tenanted is not allowed."
  https://community.atlassian.com/t5/Confluence-questions/Confluence-installation-throw-an-error-quot-Confluence-is-vacant/qaq-p/846918

Apart from that in the "Confluence startup error" thread there are some more hints and tips:

• https://community.atlassian.com/t5/Confluence-questions/Confluence-startup-error/qaq-p/624859
  
  This one for example suggests problems with tha name resolution (DNS) of the host.

The last one suggests there were some timeout issues with their reverse proxy.

• https://community.atlassian.com/t5/Confluence-questions/Confluence-is-vacant-a-call-to-tenanted-public-abstract-org/qaq-p/737863

Also, this knowledge base article might apply:
https://confluence.atlassian.com/confkb/confluence-generates-confluence-is-vacant-error-on-install-779164449.html

Please check and let us hear from you. In case you could describe further how the update was done it would help other users from community to provide a better and more detailed answer.

Cheers,
Daniel