Fatal error after upgrading from 6.9.x to 7.12 Edited

[Thorsten Reichelt]

Hi!

I tried to upgrade my Confluence server (running as docker container) from version 6.9.x to version 7.12 and something went totally wrong. :(

See atlassian-confluence.log at https://pastebin.com/Jfa81Few

After waiting ten or more minutes, my docker container just dies. The last docker log is:

[...]
log4j:ERROR Could not find value for key log4j.appender.luceneQuery
log4j:ERROR Could not instantiate appender named "luceneQuery".
15-Jun-2021 18:59:10.907 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath
15-Jun-2021 18:59:11.375 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'
18:59:11 [Catalina-utility-2] o.a.c.c.C.[.[.[/synchrony-proxy][INFO] log - Initializing Spring DispatcherServlet 'dispatcher'
2021-06-15 18:59:11,961 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.12.2 [build 8703 based on commit hash 9809c79ab645d5a5a549bafa901a5e101dc6e88c] - synchrony version 4.0.0-master-30dece1
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/hibernate.adapter-1.0.3.jar) to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Security framework of XStream not explicitly initialized, using predefined black list on your own risk.

The last messages in "atlassian-confluence.log" are:

[...]
2021-06-16 16:44:40,935 INFO [Catalina-utility-1] [confluence.upgrade.impl.DefaultUpgradeGate] waitForBooleanValue Waiting to find if plugin dependent upgrades are required. Maximum wait time will be 90 second>
2021-06-16 16:44:40,936 INFO [Catalina-utility-1] [confluence.upgrade.impl.DefaultUpgradeGate] waitForBooleanValue plugin dependent upgrades are required : false.
2021-06-16 16:44:41,617 WARN [Catalina-utility-1] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
->[com.atlassian.confluence.schedule.quartz.ManagedScheduledJobTriggerFactory.initialiseTriggers]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #944281246)
[com.atlassian.confluence.impl.schedule.caesium.SchedulerClusteredJobDao.find]: PROPAGATION_REQUIRES_NEW,ISOLATION_DEFAULT (Session #1441406646)
[com.atlassian.confluence.impl.schedule.caesium.SchedulerClusteredJobDao.delete]: PROPAGATION_REQUIRES_NEW,ISOLATION_DEFAULT (Session #2127167152)
2021-06-16 16:44:41,623 ERROR [Catalina-utility-1] [atlassian.config.lifecycle.LifecycleManager] panicAndShutdown Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:m>
java.lang.NullPointerException
at com.atlassian.confluence.schedule.quartz.ManagedScheduledJobTriggerFactory.createAndRegisterTrigger(ManagedScheduledJobTriggerFactory.java:221)
at com.atlassian.confluence.schedule.quartz.ManagedScheduledJobTriggerFactory.initialiseTriggers(ManagedScheduledJobTriggerFactory.java:152)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy201.initialiseTriggers(Unknown Source)
at com.atlassian.confluence.schedule.listeners.ManagedScheduledJobsLifecycle.startup(ManagedScheduledJobsLifecycle.java:24)
at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.startUp(DefaultLifecycleManager.java:52)
at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextInitialized(LifecycleServletContextListener.java:17)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5172)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-06-16 16:44:41,806 INFO [Caesium-1-1] [agent.service.check.StaleChecksCleaner] info Cleaning stale checks
2021-06-16 16:44:42,915 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] destroy Shutting down long running task service
2021-06-16 16:44:44,444 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

Is there any way to re-animate my confluence server?

[Brant Schroeder]

@Thorsten Reichelt Have you tried starting the application with all apps disabled?  Sometimes an app needs to update and Confluence gets stuck and can not start.  

$ ./start-confluence.sh --disable-all-addons

[Thorsten Reichelt]

Yes, I did. This makes no difference. :(

[Thorsten Reichelt]

Something must have gone very wrong with the update.
I noticed that sometimes there are "ERROR: permission denied for table XYZ" entries in the logs. After calling

for tbl in `psql -qAt -c "select tablename from pg_tables where schemaname = 'public';" confluence` ; do  psql -c "alter table \"$tbl\" owner to atlassian" confluence ; done

for tbl in `psql -qAt -c "select sequence_name from information_schema.sequences where sequence_schema = 'public';" confluence` ; do  psql -c "alter sequence \"$tbl\" owner to atlassian" confluence ; done

for tbl in `psql -qAt -c "select table_name from information_schema.views where table_schema = 'public';" confluence` ; do  psql -c "alter view \"$tbl\" owner to atlassian" confluence ; done

I managed to start confluence. But now I am stuck on an

Oops - an error has occurred
System Error

site. :(

Cause

org.apache.velocity.exception.MethodInvocationException: Invocation of method 'isShowSignUp' in class com.atlassian.confluence.user.actions.LoginAction threw exception java.lang.NullPointerException: Application cannot be null at /login.vm[line 6, column 28]
    at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:342)

caused by: java.lang.NullPointerException: Application cannot be null
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:895)

[Brant Schroeder]

@Thorsten Reichelt I would suggest opening an actual support issue with Atlassian.  

https://support.atlassian.com/contact/

[Thorsten Reichelt]

Unfortunatelly Atlassian does not want to help me: "Atlassian no longer offers personal support for Starter Licenses. Read the blog post." :( :(

[Brant Schroeder]

Are you able to roll back?  Did you make a backup prior to upgrading?