Confluence stopped working after server migration

[adrianoatl]

Our server crashed because of a hardware failure, so we have been given another VPS on a new host. We restored Confluence's files from a backup—which was a couple of weeks older than the latest one, but besides changes in the content of Confluence (i.e. in the database) nothing was changed/added in between, not even plugins.

On the database level, we restored the entire MySQL DB via dump. All is there, and should be working.

Confluence seems to be starting just fine:

To run Confluence in the foreground, start the server with start-confluence.sh -fg
executing as current user
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /usr/local/Confluence/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /usr/local/Java/jre1.8.0_161/bin/java
2018-06-24 18:04:43,148 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /usr/local/Confluence/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE:   /usr/local/Confluence/confluence
Using CATALINA_HOME:   /usr/local/Confluence/confluence
Using CATALINA_TMPDIR: /usr/local/Confluence/confluence/temp
Using JRE_HOME:        /usr/local/Java/jre1.8.0_161
Using CLASSPATH:       /usr/local/Confluence/confluence/bin/bootstrap.jar:/usr/local/Confluence/confluence/bin/tomcat-juli.jar
Using CATALINA_PID:    /usr/local/Confluence/confluence/work/catalina.pid
Tomcat started.

However, when hitting the main Confluence page, it says:

HTTP Status 500 - Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.

type Exception report

message Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.

description The server encountered an internal error that prevented it from fulfilling this request.

exception

com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
	com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)
	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
	com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
	org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)
	org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:344)
	org.springframework.orm.hibernate5.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:326)
	org.springframework.orm.hibernate5.HibernateTemplate.get(HibernateTemplate.java:443)
	org.springframework.orm.hibernate5.HibernateTemplate.get(HibernateTemplate.java:436)
	com.atlassian.confluence.user.rememberme.DefaultConfluenceRememberMeTokenDao.findById(DefaultConfluenceRememberMeTokenDao.java:21)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
	org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
	org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
	com.sun.proxy.$Proxy152.findById(Unknown Source)
	com.atlassian.seraph.service.rememberme.DefaultRememberMeService.getRememberMeCookieAuthenticatedUsername(DefaultRememberMeService.java:49)
	com.atlassian.seraph.auth.DefaultAuthenticator.getUserFromCookie(DefaultAuthenticator.java:432)
	com.atlassian.seraph.auth.DefaultAuthenticator.getUser(DefaultAuthenticator.java:332)
	com.atlassian.confluence.user.ConfluenceAuthenticator.getUser(ConfluenceAuthenticator.java:122)
	com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:139)
	com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:25)
	com.atlassian.confluence.web.filter.ThreadLocalCacheFilter.doFilter(ThreadLocalCacheFilter.java:25)
	com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter$SwapOnBootstrapFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:55)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:35)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:64)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	com.atlassian.confluence.util.ClusterHeaderFilter.doFilter(ClusterHeaderFilter.java:56)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.core.filters.cache.AbstractCachingFilter.doFilter(AbstractCachingFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:64)
	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72)
	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
	com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)

note The full stack trace of the root cause is available in the Apache Tomcat/8.0.48 logs.

---

Apache Tomcat/8.0.48

Should we perhaps request, set up and install  a new server id? Any idea on what's going on?

[AnnWorley · Accepted solution]

Hi Adriano,

Please make sure the new host can resolve it's host name to an IP address as described in Confluence generates Confluence is vacant error on install.

You mentioned you restored files and the database, please note that the confluence_home  should be included in the files from the old instance, as mentioned in Migrating Confluence Between Servers

Other customers have reported that the error you posted can be caused by Confluence connecting to a database server that is not quite ready:

I've seen this caused by two things in the past.

One is that the database is not ready to serve at the time Confluence needs it. 

Looking forward to hearing if the name resolution is the issue or perhaps the database server.

Thanks,

Ann

[adrianoatl]

Thanks a lot, Ann, for taking the time to send me all these pointers. It is indeed much appreciated.

The issue was actually with the database; some rights had not been created/restored properly.

Everything is now working just fine.

[AnnWorley]

That's great news, thanks for circling back to let us know. :)