Upgrade to 7.x throws 500 errors on all pages

[Andrew Smith]

Hi guys,

I've spent the entire weekend trying to upgrade from 6.15.9 to 7.0.1 and no matter which way I go about it, it seems to upgrade the database over several minutes, then every page load (login.action, authenticate.action, pulp-render.action) return 500 errors.

It shows absolutely nothing in catalina.out

How do I progress with debugging this issue, as I obviously can't get to the control panel to extract a support payload either?

Thanks,

Andrew

[Ram Bhusal]

We had similar issues after reboot. The following actions could help.

1. check MySQL or PostgreSQL is up and running. 
2. check httpd is up and running. ---- This is what was missing on my end.
   1. Had done additional steps since this was not up after reboot.
      1. Check'
         1. systemctl is-enabled httpd
         2. service httpd status
      2. To Enable
         1. systemctl enable httpd
      3. check and start httpd if not running.
3. ping domain to ensure you are connected to network.
4. final step start confluence

Hope this helps

Good Luck. :)

[Nic Brough -Adaptavist-]

What does the application log say the problem is?  (Jira home/logs/atlassian-jira.log is the main one although others in that directory might tell you something too)

Have you changed the old location of Jira in the server.xml with the "context" directive?  has that been replaced in the server.xml file during upgrade?

[Andrew Smith]

Yep all empty.

Tried upgrading with an unchanged server.xml, as well as trying with the modified (nginx https proxy), both failed and threw 500's no matter which page I navigated to

[Nic Brough -Adaptavist-]

If the application log is empty, then the service is not running.  If the service starts and then halts without writing to the application log, then there will be an error written into the catalina.out, or an error thrown on to screen when you run the start up script.

[Andrew Smith]

I'll try again right now, sec.

[Andrew Smith]

Hi Nic,

I have upgraded to 7.0.1 once again, the atlassian-jira.log is now populated, I seem to be receiving "every plugin we have" throwing something that looks very similar to this

2019-09-29 11:11:59,281 ERROR [ThreadPoolAsyncTaskExecutor::Thread 1] [internal.dependencies.startup.DependencyWaiterApplicationContextExecutor] fail Unable to create application context for [com.gliffy.plugin.confluence.gliffy-confluence-plugin], unsatisfied dependencies: none
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'diagramManager' defined in URL [bundle://290.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.gliffy.plugin.confluence.diagram.ConfluenceDiagramManager]: Constructor threw exception; nested exception is java.lang.NullPointerException

[Andrew Smith]

2019-09-29 11:06:50,917 ERROR [Catalina-utility-1] [plugin.osgi.factory.OsgiPlugin] enableInternal Detected an error (BundleException) enabling the plugin 'com.atlassian.confluence.plugins.confluence-questions' : Unable to resolve com.atlassian.confluence.plugins.confluence-questions [267](R 267.0): missing requirement [com.atlassian.confluence.plugins.confluence-questions [267](R 267.0)] osgi.wiring.package; (&(osgi.wiring.package=com.atlassian.plugin.util.collect)(version>=3.0.7)) Unresolved requirements: [[com.atlassian.confluence.plugins.confluence-questions [267](R 267.0)] osgi.wiring.package; (&(osgi.wiring.package=com.atlassian.plugin.util.collect)(version>=3.0.7))]. This error usually occurs when your plugin imports a package from another bundle with a specific version constraint and either the bundle providing that package doesn't meet those version constraints, or there is no bundle available that provides the specified package. For more details on how to fix this, see https://developer.atlassian.com/x/mQAN
2019-09-29 11:06:50,917 WARN [Catalina-utility-1] [atlassian.plugin.impl.AbstractPlugin] enable Unable to enable plugin 'com.atlassian.confluence.plugins.confluence-questions'
2019-09-29 11:06:50,918 WARN [Catalina-utility-1] [atlassian.plugin.impl.AbstractPlugin] enable Because of this exception
com.atlassian.plugin.osgi.container.OsgiContainerException: Cannot start plugin: com.atlassian.confluence.plugins.confluence-questions
at com.atlassian.plugin.osgi.factory.OsgiPlugin.enableInternal(OsgiPlugin.java:424)
at com.atlassian.plugin.impl.AbstractPlugin.enable(AbstractPlugin.java:252)
at com.atlassian.plugin.manager.PluginEnabler.actualEnable(PluginEnabler.java:120)
at com.atlassian.plugin.manager.PluginEnabler.enable(PluginEnabler.java:97)
at com.atlassian.plugin.manager.DefaultPluginManager.lambda$enableDependentPlugins$22(DefaultPluginManager.java:1246)
at com.atlassian.plugin.manager.PluginTransactionContext.wrap(PluginTransactionContext.java:63)
at com.atlassian.plugin.manager.DefaultPluginManager.enableDependentPlugins(DefaultPluginManager.java:1230)
at com.atlassian.plugin.manager.DefaultPluginManager.lambda$addPlugins$20(DefaultPluginManager.java:1215)
at com.atlassian.plugin.manager.PluginTransactionContext.wrap(PluginTransactionContext.java:63)
at com.atlassian.plugin.manager.DefaultPluginManager.addPlugins(DefaultPluginManager.java:1115)
at com.atlassian.plugin.manager.DefaultPluginManager.lambda$lateStartup$4(DefaultPluginManager.java:652)
at com.atlassian.plugin.manager.PluginTransactionContext.wrap(PluginTransactionContext.java:63)
at com.atlassian.plugin.manager.DefaultPluginManager.lateStartup(DefaultPluginManager.java:645)
at com.atlassian.confluence.plugin.ConfluencePluginManager.lateStartup(ConfluencePluginManager.java:149)
at com.atlassian.confluence.plugin.PluginFrameworkContextListener.contextInitialized(PluginFrameworkContextListener.java:81)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4682)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5143)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

[Andrew Smith]

Caused by: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getConfluenceResourceTags' in class com.atlassian.confluence.setup.velocity.VelocityFriendlyPageBuilderService threw exception com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException at /decorators/includes/header.vm[line 92, column 24]
at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:342)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:284)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:262)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:342)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
at org.apache.velocity.runtime.directive.Parse.render(Parse.java:263)
at com.atlassian.confluence.setup.velocity.ProfilingParseDirective.render(ProfilingParseDirective.java:21)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
at org.apache.velocity.Template.merge(Template.java:328)
at org.apache.velocity.Template.merge(Template.java:235)
at com.atlassian.confluence.util.velocity.VelocityUtils.renderTemplateWithoutSwallowingErrors(VelocityUtils.java:70)
at com.atlassian.confluence.util.velocity.VelocityUtils.renderTemplateWithoutSwallowingErrors(VelocityUtils.java:76)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator$StreamingResponseWritingStrategy.renderToResponse(ConfluenceSitemeshDecorator.java:271)
... 81 more

[Andrew Smith]

The log file seems to now be on a spam loop (hundreds) with the error above.

[Andrew Smith]

2019-09-29 11:37:10,529 INFO [http-nio-8090-exec-8] [atlassian.confluence.status.SystemErrorInformationLogger] writeToLog
Request Unique ID : 5ea50e0b-f81f-431d-b911-81f428d34256
--------------------------
JVM Stats
--------------------------
allocatedHeap = 1073741824
freeAllocatedHeap = 486186568
availablePermGen = 0
maxPermGen = -1
availableHeap = 486186568
maxHeap = 1073741824
usedHeap = 587555256
usedPermGen = -1
--------------------------
Request Information
--------------------------
URL: https://confluence.heffron.com.au/500page.jsp
Scheme: https
Server: confluence.heffron.com.au
Port: 443
URI: /500page.jsp
Context Path:
Servlet Path: /500page.jsp
Path Info: null
Query String: null
--------------------------
Attributes
--------------------------
javax.servlet.forward.request_uri: /
javax.servlet.forward.context_path:
javax.servlet.forward.servlet_path: /index.action
javax.servlet.forward.mapping: org.apache.catalina.core.ApplicationMapping$MappingImpl@404d8de2
javax.servlet.error.status_code: 500
com.atlassian.confluence.impl.profiling.DecoratorTimings: com.atlassian.confluence.impl.profiling.DecoratorTimings@7a6890f
com.opensymphony.sitemesh.APPLIED_ONCE: true
__cleanup_recursion_counter: 0
webwork.request_uri: /
atlas.webInterfaceContext: com.atlassian.confluence.plugin.descriptor.web.DefaultWebInterfaceContext@4cb3d914
com.atlassian.confluence.util.message.MessagesDecoratorFilter__already_filtered__: true
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter_already_filtered: true
atlassian.core.seraph.original.url: /500page.jsp
theme.resource.path: /download/resources/com.refinedwiki.confluence.plugins.theme.original%3Amain/
com.atlassian.labs.botkiller.BotKillerFilter: true
com.atlassian.gzipfilter.GzipFilter_already_filtered: true
Confluence-Request-Time: 1569757030075
com.atlassian.confluence.web.ConfluenceJohnsonFilter_already_filtered: true
com.atlassian.seraph.auth.LoginReason: OK
__sitemesh__page: com.opensymphony.sitemesh.compatability.Content2HTMLPage@60731d0d
webwork.valueStack: com.opensymphony.xwork.util.OgnlValueStack@4c7dae7c
confluence.themecontext: com.atlassian.confluence.themes.ThemeContext@7f20b80d
webwork.view_uri: /dashboard.action
javax.servlet.error.message:
javax.servlet.error.servlet_name: action
com.refinedwiki.confluence.plugins.theme.original.filters.StartSiteFilter_already_filtered: true
com.opensymphony.sitemesh.USINGSTREAM: false
com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator: true
B3-TraceId: c5a569a595dd72
loginfilter.already.filtered: true
javax.servlet.error.request_uri: /
com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true
javax.servlet.error.exception: java.lang.RuntimeException: javax.servlet.ServletException: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getConfluenceResourceTags' in class com.atlassian.confluence.setup.velocity.VelocityFriendlyPageBuilderService threw exception com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException at /decorators/includes/header.vm[line 92, column 24]
os_securityfilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true
--------------------------
Parameters
--------------------------
caused by: java.lang.RuntimeException: javax.servlet.ServletException: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getConfluenceResourceTags' in class com.atlassian.confluence.setup.velocity.VelocityFriendlyPageBuilderService threw exception com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException at /decorators/includes/header.vm[line 92, column 24]
at com.opensymphony.sitemesh.webapp.decorator.BaseWebAppDecorator.render(BaseWebAppDecorator.java:37)
caused by: javax.servlet.ServletException: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getConfluenceResourceTags' in class com.atlassian.confluence.setup.velocity.VelocityFriendlyPageBuilderService threw exception com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException at /decorators/includes/header.vm[line 92, column 24]
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator$ErrorSendingErrorHandlingStrategy.handleException(ConfluenceSitemeshDecorator.java:224)
caused by: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getConfluenceResourceTags' in class com.atlassian.confluence.setup.velocity.VelocityFriendlyPageBuilderService threw exception com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException at /decorators/includes/header.vm[line 92, column 24]
at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:342)
caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
caused by: java.lang.NullPointerException
at com.comalatech.confluence.adhoclists.condition.ConfluenceBuildNumberCondition.shouldDisplay(ConfluenceBuildNumberCondition.java:25)

2019-09-29 11:37:10,601 WARN [http-nio-8090-exec-8] [webresource] evaluateSafely exception thrown in `shouldDisplayImmediate` during condition evaluation
-- traceId: 39f48d4edf18096f
java.lang.NullPointerException
at com.comalatech.confluence.adhoclists.condition.ConfluenceBuildNumberCondition.shouldDisplay(ConfluenceBuildNumberCondition.java:25)
at com.atlassian.plugin.webresource.condition.DecoratingLegacyCondition.shouldDisplayImmediate(DecoratingLegacyCondition.java:53)
at com.atlassian.plugin.webresource.impl.CachedCondition.evaluateSafely(CachedCondition.java:49)
at com.atlassian.plugin.webresource.impl.helpers.UrlGenerationHelpers.buildIncludedExcludedConditionsAndBatchingOptions(UrlGenerationHelpers.java:248)
at com.atlassian.plugin.webresource.impl.helpers.UrlGenerationHelpers.resolve(UrlGenerationHelpers.java:204)
at com.atlassian.plugin.webresource.assembler.DefaultWebResourceAssembler$1.resolve(DefaultWebResourceAssembler.java:87)
at com.atlassian.plugin.webresource.assembler.DefaultWebResourceAssembler$1.drainOrPoll(DefaultWebResourceAssembler.java:76)
at com.atlassian.plugin.webresource.assembler.DefaultWebResourceAssembler$1.drainIncludedResources(DefaultWebResourceAssembler.java:67)
at com.atlassian.confluence.plugin.webresource.DefaultConfluenceWebResourceService.writeConfluenceResourceTags(DefaultConfluenceWebResourceService.java:129)
at com.atlassian.confluence.setup.velocity.VelocityFriendlyPageBuilderService.getConfluenceResourceTags(VelocityFriendlyPageBuilderService.java:104)
at sun.reflect.GeneratedMethodAccessor1015.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:385)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:374)
at com.atlassian.velocity.htmlsafe.introspection.AnnotationBoxingMethod.invoke(AnnotationBoxingMethod.java:24)
at com.atlassian.velocity.htmlsafe.introspection.UnboxingMethod.invoke(UnboxingMethod.java:28)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:270)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:262)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:342)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
at org.apache.velocity.runtime.directive.Parse.render(Parse.java:263)
at com.atlassian.confluence.setup.velocity.ProfilingParseDirective.render(ProfilingParseDirective.java:21)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
at org.apache.velocity.Template.merge(Template.java:328)
at org.apache.velocity.Template.merge(Template.java:235)
at com.atlassian.confluence.util.velocity.VelocityUtils.renderTemplateWithoutSwallowingErrors(VelocityUtils.java:70)
at com.atlassian.confluence.util.velocity.VelocityUtils.renderTemplateWithoutSwallowingErrors(VelocityUtils.java:76)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator$StreamingResponseWritingStrategy.renderToResponse(ConfluenceSitemeshDecorator.java:271)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator.applyDecoratorUsingVelocity(ConfluenceSitemeshDecorator.java:198)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator.applyDecorator(ConfluenceSitemeshDecorator.java:138)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshErrorDecorator.renderInternal(ConfluenceSitemeshErrorDecorator.java:44)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator.lambda$render$0(ConfluenceSitemeshDecorator.java:100)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshDecorator.render(ConfluenceSitemeshDecorator.java:98)
at com.atlassian.confluence.util.profiling.ConfluenceSitemeshErrorDecorator.render(ConfluenceSitemeshErrorDecorator.java:36)
at com.opensymphony.sitemesh.webapp.decorator.BaseWebAppDecorator.render(BaseWebAppDecorator.java:33)
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:84)
at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:56)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:43)
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:242)
at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.applyFilter(ConfluenceSecurityFilter.java:40)
at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:29)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:56)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:43)
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:56)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:43)
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)
at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:348)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

[Nic Brough -Adaptavist-]

Ok, have you upgraded all the apps as well as the core?  The errors are happening during the load of the add-ons, so I suspect the existing ones are out of date and crashing when loading.

If you can't get to them, then try to restart it without them.  Stop Confluence, go to <confluence home>/plugins and find all the .jar files in there.  Move them somewhere safe and outside Confluence (like a new directory in your home directory) and restart it, this should enable Confluence to start up ok, then you can use "manage add ons" to add the later versions back in (it will retain the data from the old ones)

[Andrew Smith]

Yep, definitely updated all the plugins to the latest prior to the upgrade.

Strangely there is no plugins directory here, nor is there on the datacentre shared home. The bundled-plugins folder is empty.

[root@confluence01 confluence]# pwd
/var/atlassian/application-data/confluence
[root@confluence01 confluence]# ls -lha
total 60M
drwx------ 21 confluence confluence 4.0K Sep 29 14:06 .
drwxr-xr-x 3 confluence confluence 4.0K Sep 29 04:05 ..
drwx------ 2 confluence confluence 4.0K Mar 22 2018 analytics-logs
drwx------ 3 confluence confluence 4.0K Mar 22 2018 attachments
drwxr-x--- 2 confluence confluence 4.0K Sep 29 12:45 bundled-plugins
-rw-r----- 1 confluence confluence 6.2K Sep 29 14:06 confluence.cfg.xml
drwx------ 2 confluence confluence 4.0K Aug 29 2017 export
drwx------ 2 confluence confluence 4.0K Feb 2 2018 imgEffects
drwx------ 4 confluence confluence 12K Sep 30 03:42 index
drwx------ 2 confluence confluence 4.0K Sep 30 03:42 journal
-rw------- 1 confluence confluence 0 Aug 14 2017 lock
drwx------ 3 confluence confluence 4.0K Sep 29 12:12 logs
drwx------ 5 confluence confluence 4.0K Mar 22 2018 originaltheme
drwxr-x--- 2 confluence confluence 4.0K Sep 29 14:11 plugins-cache
drwxr-x--- 5 confluence confluence 4.0K Sep 29 12:45 plugins-osgi-cache
drwxr-x--- 2 confluence confluence 4.0K Sep 29 12:45 plugins-temp
drwx------ 2 confluence confluence 4.0K Mar 22 2018 recovery
drwx------ 21 confluence confluence 4.0K Sep 30 00:05 sandbox
drwx------ 5 confluence confluence 4.0K Mar 22 2018 shared-home
-rw-r----- 1 confluence confluence 60M Sep 29 14:05 synchrony-standalone.jar
drwx------ 2 confluence confluence 16K Sep 30 02:01 temp
drwx------ 14 confluence confluence 4.0K Mar 22 2018 thumbnails
drwx------ 3 confluence confluence 4.0K Feb 2 2018 viewfile
drwx------ 2 confluence confluence 20K Sep 30 03:42 webresource-temp

[Nick Mancuso]

For those like me who came to this exact post only to not find the solution... here it is!

If you are running Confluence 6.1 and later you can run the following to start confluence up without any plugins

1. stop you confluence instance

2. go to your /opt/atlassian/confluence/bin folder and run

./start-confluence.sh --disable-all-addons

When it starts up you can then go into the admin interface and see which plugins were causing the error

More details about disabling can be found here: https://confluence.atlassian.com/confkb/how-to-manually-remove-malfunctioning-add-ons-113705118.html

[Pavel Junek]

Hi @Andrew Smith,

Welcome to Atlassian community!

I will try help you with your problem.

• Which database using? 
• Is something written in {confluence-home}/log/atlassian-confluence.log file?
• Did you keep the upgrade running long enough?

Thank you!

Pavel

[Andrew Smith]

Hi there,

Yes ran the upgrade for 6 hours. top showed Java using < 1% cpu by then.

atlassian-confluence.log empty

postgresql

[Pavel Junek]

I'm still thinking - do you have folder permissions set to confluence:confluence for {cofluence-install} and {confluence-home} folder?

[Andrew Smith]

Most certainly do yes