Upgrade from Confluence Server 7.1.2 to 7.5.1 fails. 500 Error on Startup

Andrew Mullen_bbky3j

Before I get too deep into the details, going to note that I've already done a bunch of research and found some other folks have had similar problems.  This is a very standard install, no plugins, no 3rd party anything, for a small Customer Support group.  We have 10 users, and maybe a thousand pages.

Running on MSSQL DB, Windows 10 computer.  Running as bundled JRE startup instead of as a Service. 

>echo %java_home%  -->  C:\Progra~1\Java\jdk-11.0.5

I already looked online and found some workarounds, none of which worked for me.
Here's what happened:

1. I am following a standard upgrade path found here:  https://confluence.atlassian.com/doc/upgrading-confluence-4578.html
2. Ran into an error during upgrade:  com.install4j.runtime.beans.actions.control.Runscriptaction failed
3. According to various notes, this is something that could be ignored and fixed later:
   1. Article 1:  https://community.atlassian.com/t5/Confluence-questions/Cannot-upgrade-Confluence-from-7-2-2-to-7-3-or-7-4/qaq-p/1357607
   2. Article 2:  https://jira.atlassian.com/browse/CONFSERVER-59387 (left a comment on this bug)
4. Finished the update, then tried to start Confluence, but it wouldn't start.  Always got a 500 Error.
5. I reverted back to the working version, and tried again.  Still same error.
6. None of the workarounds noted in any of the articles I could find would work. 
7. I tried changing permissions on files and directories, removing plugin cache, and more.

I ended up reverting to my existing version 7.1.2 and that's at least functional, but I would like to upgrade to 7.5.1 and can't seem to make it work.  I'd prefer to do this as an upgrade rather than as a fresh install, I just worry about having to reconfigure my SQL settings on a clean install. 

Last message from Catalina.out copied below:

11-Jun-2020 17:01:52.548 SEVERE [http-nio-8090-exec-1] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
 org.apache.jasper.JasperException: An exception occurred processing [/500page.jsp] at line [120]117:
118:          if (sysInfoService != null)
119:          {
120:           confluenceInfo = sysInfoService.getConfluenceInfo();
121:           memoryInfo = sysInfoService.getMemoryInfo();
122:           dbInfo = sysInfoService.getSafeDatabaseInfo();
123:                 sysinfo = GeneralUtil.convertBeanToMap(sysInfoService.getSystemProperties());
Stacktrace:
  at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:626)
  at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:510)
  at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
  at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
  at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
  at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
  at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.HttpSessionRegistrarFilter.doFilter(HttpSessionRegistrarFilter.java:44)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:67)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
  at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
  at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
  at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
  at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
  at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)
  at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)
  at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:348)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.base/java.lang.Thread.run(Unknown Source)
 Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
  at com.atlassian.confluence.impl.tenant.TenantedProxyFactory.lambda$throwVacantExceptionOnMethodCall$3(TenantedProxyFactory.java:98)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  at com.sun.proxy.$Proxy68.getCurrentSession(Unknown Source)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.base/java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
  at com.sun.proxy.$Proxy68.getCurrentSession(Unknown Source)
  at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:370)
  at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:337)
  at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:48)
  at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.base/java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  at com.sun.proxy.$Proxy70.retrieve(Unknown Source)
  at com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.lambda$retrieve$0(ReadThroughCachingBandanaPersister.java:84)
  at com.atlassian.confluence.impl.cache.ReadThroughAtlassianCache.lambda$get$0(ReadThroughAtlassianCache.java:76)
  at com.atlassian.confluence.cache.ehcache.DefaultConfluenceEhCache$ValueWrapperSupplier.get(DefaultConfluenceEhCache.java:228)
  at com.atlassian.confluence.cache.ehcache.DefaultConfluenceEhCache$ValueWrapperSupplier.get(DefaultConfluenceEhCache.java:217)
  at com.atlassian.cache.ehcache.ReferenceCacheLoader.load(ReferenceCacheLoader.java:31)
  at com.atlassian.cache.ehcache.wrapper.ValueProcessorEhcacheLoaderDecorator.load(ValueProcessorEhcacheLoaderDecorator.java:30)
  at net.sf.ehcache.Cache.loadValueUsingLoader(Cache.java:3360)
  at net.sf.ehcache.Cache.getWithLoader(Cache.java:1848)
  at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:128)
  at com.atlassian.confluence.cache.ehcache.DefaultConfluenceEhCache.get(DefaultConfluenceEhCache.java:47)
  at com.atlassian.confluence.cache.ConfluenceMonitoringCache.get(ConfluenceMonitoringCache.java:57)
  at com.atlassian.confluence.impl.cache.ReadThroughAtlassianCache.get(ReadThroughAtlassianCache.java:88)
  at com.atlassian.confluence.impl.cache.ReadThroughCache.get(ReadThroughCache.java:28)
  at com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.retrieve(ReadThroughCachingBandanaPersister.java:82)
  at com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.retrieve(ReadThroughCachingBandanaPersister.java:75)
  at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
  at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
  at com.atlassian.confluence.setup.settings.DefaultSettingsManager.getGlobalSettings(DefaultSettingsManager.java:40)
  at com.atlassian.confluence.status.service.DefaultSystemInformationService.getConfluenceInfo(DefaultSystemInformationService.java:114)
  at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:309)
  at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:71)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
  at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477)
  ... 65 more

Shannon S

Hello Andrew,

Welcome to Atlassian Community! It's nice to have you join us. I'm sorry that your upgrade isn't running as smoothly for you as we'd hope. I'm happy to help you get back online.

The stack trace that you provided from CATALINA.OUT is due to the following issue: (Confluence generates Confluence is vacant error on install) 

Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.

However, more detailed error logs specific to your issue are most likely in your atlassian-confluence.log file. See Working with Confluence Logs for more details.

Next Steps

1. You mentioned that you followed the standard upgrade path in Upgrading Confluence. Can you confirm if that means you used the installer method over the manual method? If so, you want to make sure that you ran the installer with a Windows administrator account. This will ensure that all the right files are being replaced. If that is the case, you can review the Confluence is vacant error page to ensure that Confluence is able to reach itself, as well as its database.
2. If you happen to see any C3P0PooledConnectionPoolManager errors, then that would indicate Confluence cannot reach the database, which would also cause the vacant error. In that situation, you'll want to make sure that your <confluence-home>/confluence.cfg.xml file has the right configuration to connect to the database. You also want to check that your server.xml file contains accurate information.

3. Make sure any Confluence or Synchrony processes are killed in Task Manager. If the account running the script has the correct permissions, you can manually start Confluence with the correct admin account using the <installation-directory>\bin\start-confluence.bat script, provided it's not being run as a service.

4. Running the BAT, the DOS output will tell you if there are any issues starting Confluence. For example, Confluence could be starting faster than the database service, and then becomes vacant, since the database isn't available yet. In this case, you'd want to make sure the database service is up and running before manually starting Confluence.

5. Clearing the plugins cache prior to starting the BAT file will help cleanup any inconsistent state preventing boot.

If you're still encountering issues after doing these things, please have a look at your atlassian-confluence.log file upon restart. Feel free to send any relevant log files and we can have a look.

Shannon