Upgrade from Confluence Server 7.1.2 to 7.5.1 fails. 500 Error on Startup

[Andrew Mullen]

Before I get too deep into the details, going to note that I've already done a bunch of research and found some other folks have had similar problems.  This is a very standard install, no plugins, no 3rd party anything, for a small Customer Support group.  We have 10 users, and maybe a thousand pages.

Running on MSSQL DB, Windows 10 computer.  Running as bundled JRE startup instead of as a Service. 

>echo %java_home%  -->  C:\Progra~1\Java\jdk-11.0.5

I already looked online and found some workarounds, none of which worked for me.
Here's what happened:

1. I am following a standard upgrade path found here:  https://confluence.atlassian.com/doc/upgrading-confluence-4578.html
2. Ran into an error during upgrade:  com.install4j.runtime.beans.actions.control.Runscriptaction failed
3. According to various notes, this is something that could be ignored and fixed later:
   1. Article 1:  https://community.atlassian.com/t5/Confluence-questions/Cannot-upgrade-Confluence-from-7-2-2-to-7-3-or-7-4/qaq-p/1357607
   2. Article 2:  https://jira.atlassian.com/browse/CONFSERVER-59387 (left a comment on this bug)
4. Finished the update, then tried to start Confluence, but it wouldn't start.  Always got a 500 Error.
5. I reverted back to the working version, and tried again.  Still same error.
6. None of the workarounds noted in any of the articles I could find would work. 
7. I tried changing permissions on files and directories, removing plugin cache, and more.

I ended up reverting to my existing version 7.1.2 and that's at least functional, but I would like to upgrade to 7.5.1 and can't seem to make it work.  I'd prefer to do this as an upgrade rather than as a fresh install, I just worry about having to reconfigure my SQL settings on a clean install. 

Last message from Catalina.out copied below:

11-Jun-2020 17:01:52.548 SEVERE [http-nio-8090-exec-1] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
 org.apache.jasper.JasperException: An exception occurred processing [/500page.jsp] at line [120]117:
118:          if (sysInfoService != null)
119:          {
120:           confluenceInfo = sysInfoService.getConfluenceInfo();
121:           memoryInfo = sysInfoService.getMemoryInfo();
122:           dbInfo = sysInfoService.getSafeDatabaseInfo();
123:                 sysinfo = GeneralUtil.convertBeanToMap(sysInfoService.getSystemProperties());
Stacktrace:
  at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:626)
  at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:510)
  at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
  at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
  at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
  at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
  at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.HttpSessionRegistrarFilter.doFilter(HttpSessionRegistrarFilter.java:44)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:67)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
  at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
  at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
  at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
  at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
  at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)
  at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)
  at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:348)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.base/java.lang.Thread.run(Unknown Source)
 Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
  at com.atlassian.confluence.impl.tenant.TenantedProxyFactory.lambda$throwVacantExceptionOnMethodCall$3(TenantedProxyFactory.java:98)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  at com.sun.proxy.$Proxy68.getCurrentSession(Unknown Source)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.base/java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
  at com.sun.proxy.$Proxy68.getCurrentSession(Unknown Source)
  at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:370)
  at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:337)
  at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:48)
  at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.base/java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  at com.sun.proxy.$Proxy70.retrieve(Unknown Source)
  at com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.lambda$retrieve$0(ReadThroughCachingBandanaPersister.java:84)
  at com.atlassian.confluence.impl.cache.ReadThroughAtlassianCache.lambda$get$0(ReadThroughAtlassianCache.java:76)
  at com.atlassian.confluence.cache.ehcache.DefaultConfluenceEhCache$ValueWrapperSupplier.get(DefaultConfluenceEhCache.java:228)
  at com.atlassian.confluence.cache.ehcache.DefaultConfluenceEhCache$ValueWrapperSupplier.get(DefaultConfluenceEhCache.java:217)
  at com.atlassian.cache.ehcache.ReferenceCacheLoader.load(ReferenceCacheLoader.java:31)
  at com.atlassian.cache.ehcache.wrapper.ValueProcessorEhcacheLoaderDecorator.load(ValueProcessorEhcacheLoaderDecorator.java:30)
  at net.sf.ehcache.Cache.loadValueUsingLoader(Cache.java:3360)
  at net.sf.ehcache.Cache.getWithLoader(Cache.java:1848)
  at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:128)
  at com.atlassian.confluence.cache.ehcache.DefaultConfluenceEhCache.get(DefaultConfluenceEhCache.java:47)
  at com.atlassian.confluence.cache.ConfluenceMonitoringCache.get(ConfluenceMonitoringCache.java:57)
  at com.atlassian.confluence.impl.cache.ReadThroughAtlassianCache.get(ReadThroughAtlassianCache.java:88)
  at com.atlassian.confluence.impl.cache.ReadThroughCache.get(ReadThroughCache.java:28)
  at com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.retrieve(ReadThroughCachingBandanaPersister.java:82)
  at com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.retrieve(ReadThroughCachingBandanaPersister.java:75)
  at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
  at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
  at com.atlassian.confluence.setup.settings.DefaultSettingsManager.getGlobalSettings(DefaultSettingsManager.java:40)
  at com.atlassian.confluence.status.service.DefaultSystemInformationService.getConfluenceInfo(DefaultSystemInformationService.java:114)
  at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:309)
  at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:71)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
  at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477)
  ... 65 more

[Shannon S]

Hello Andrew,

Welcome to Atlassian Community! It's nice to have you join us. I'm sorry that your upgrade isn't running as smoothly for you as we'd hope. I'm happy to help you get back online.

The stack trace that you provided from CATALINA.OUT is due to the following issue: (Confluence generates Confluence is vacant error on install) 

Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.

However, more detailed error logs specific to your issue are most likely in your atlassian-confluence.log file. See Working with Confluence Logs for more details.

Next Steps

1. You mentioned that you followed the standard upgrade path in Upgrading Confluence. Can you confirm if that means you used the installer method over the manual method? If so, you want to make sure that you ran the installer with a Windows administrator account. This will ensure that all the right files are being replaced. If that is the case, you can review the Confluence is vacant error page to ensure that Confluence is able to reach itself, as well as its database.
2. If you happen to see any C3P0PooledConnectionPoolManager errors, then that would indicate Confluence cannot reach the database, which would also cause the vacant error. In that situation, you'll want to make sure that your <confluence-home>/confluence.cfg.xml file has the right configuration to connect to the database. You also want to check that your server.xml file contains accurate information.

3. Make sure any Confluence or Synchrony processes are killed in Task Manager. If the account running the script has the correct permissions, you can manually start Confluence with the correct admin account using the <installation-directory>\bin\start-confluence.bat script, provided it's not being run as a service.

4. Running the BAT, the DOS output will tell you if there are any issues starting Confluence. For example, Confluence could be starting faster than the database service, and then becomes vacant, since the database isn't available yet. In this case, you'd want to make sure the database service is up and running before manually starting Confluence.

5. Clearing the plugins cache prior to starting the BAT file will help cleanup any inconsistent state preventing boot.

If you're still encountering issues after doing these things, please have a look at your atlassian-confluence.log file upon restart. Feel free to send any relevant log files and we can have a look.

Shannon

[Andrew Mullen]

Hello @Shannon S , I ended up finding the problem buried in the catalina logs, it was that TLS 1.0 is no longer supported for my jdbc connection, and even after adding the variable to add it as usable in one of the configuration files, Confluence still wouldn't start up.  After looking at some other articles in regards to this, I found that my issue was that I was still using SQL Server 2008 R2 and needed up upgrade that to SQL Server 2016 at least.  

I did that, then the upgrade was more smooth.  That said, during installation of the latest Confluence, I still ran into CONFSERVER-59387 but that didn't seem to be a problem.  I didn't need to make any other changes. 

So in the end, the main points that were causing my trouble were:

1. Needed to upgrade from Java JDK 8 to JDK 11 before installation.
2. Needed to upgrade from SQL Server 2008 R2 to SQL Server 2016.  

Once both those conditions were met, I was able to get Confluence 7.1.2 upgraded to Confluence 7.5.1 and it started up using the bundled JRE startup right away.  

Thank you for the kind reply!  I'm not sure how to close this ticket out though.  

[Shannon S]

Hello Andrew,

Thank you for following-up with me to let me know you were able to solve the issue and upgrade Confluence. That's great news!

To close the case, you can add an answer separate from mine in the box below labeled Suggest an answer, and I can accept your solution. Otherwise, you can just accept my reply to you above, and the question will be marked as resolved.

Take care, and have a pleasant weekend.

Shannon