Confluence don't start after update from 7.11.1 to 7.12.2

[jblanco]

I have just updated my Confluence server and doesn't boot. I really like Confluence but the latest updates are crap and Atlassian doesn't even provide any support.

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2021-06-24_14-37-46.log instead.
24-Jun-2021 14:37:46.782 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server] failed to set property [debug] to [0]
24-Jun-2021 14:37:46.822 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]
24-Jun-2021 14:37:46.832 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]
24-Jun-2021 14:37:46.835 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]
24-Jun-2021 14:37:46.859 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
24-Jun-2021 14:37:46.878 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
24-Jun-2021 14:37:47.071 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
24-Jun-2021 14:37:47.090 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [377] milliseconds
24-Jun-2021 14:37:47.094 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
24-Jun-2021 14:37:47.094 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.45]
24-Jun-2021 14:37:47.833 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
log4j:ERROR Could not find value for key log4j.appender.luceneQuery
log4j:ERROR Could not instantiate appender named "luceneQuery".
24-Jun-2021 14:37:47.992 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath
24-Jun-2021 14:37:48.148 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'
14:37:48 [Catalina-utility-2] o.a.c.c.C.[.[.[/synchrony-proxy][INFO] log - Initializing Spring DispatcherServlet 'dispatcher'
2021-06-24 14:37:48,236 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.12.2 [build 8703 based on commit hash 9809c79ab645d5a5a549bafa901a5e101dc6e88c] - synchrony version 4.0.0-master-30dece19
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/hibernate.adapter-1.0.3.jar) to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Security framework of XStream not explicitly initialized, using predefined black list on your own risk.
Exception in thread "I/O dispatcher 3" Exception in thread "I/O dispatcher 7" Exception in thread "I/O dispatcher 10" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 6"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 8"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 7"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "C3P0PooledConnectionPoolManager[identityToken->2t7qwqaihbuy1pl79x0|3af04c8f]-AdminTaskTimer"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 4"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncFileHandlerWriter-234698513"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "diagnostics-macro-rendering-thread"
Exception in thread "httpclient-io:thread-1" Exception in thread "logback-5" Exception in thread "ThreadPoolAsyncTaskExecutor::Thread 19" Exception in thread "I/O dispatcher 5" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 9" java.lang.OutOfMemoryError: Java heap space
Exception in thread "logback-7" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "C3P0PooledConnectionPoolManager[identityToken->2t7qwqaihbuy1pl79x0|3af04c8f]-HelperThread-#2" java.lang.OutOfMemoryError: Java heap space
Exception in thread "C3P0PooledConnectionPoolManager[identityToken->2t7qwqaihbuy1pl79x0|3af04c8f]-HelperThread-#1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 20" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 17" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 19" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 15" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 16" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 12" java.lang.OutOfMemoryError: Java heap space
Exception in thread "push-notification-http-client-io:thread-1" java.lang.OutOfMemoryError: Java heap space

[cruise]

How to modify the memory

[Nic Brough -Adaptavist-]

These errors are very clear - you've run out of memory for the process.  Confluence is not light on memory (few Java applications are in my opinion), but an upgrade from 7.11 to 7.12 really shouldn't be chewing up more memory.

What memory settings is this actually ending up running with?

Easiest way to tell is if it's still running is to run "ps -ef | grep -i java" on the command line (apologies if you're on windows, I can't tell from your logs) and look for variables containing the word MEMORY in the three or four lines that show the running process.  If you don't see memory, look for Xmx instead.  What are the numbers for those variables?

[jblanco]

Java process dies after running the confluence start script , so the ps won't provide useful information. Thank you anyway for your answer. As I take a backup of everything every day I will make a rollback to version 7.11.1.

[Nic Brough -Adaptavist-]

Ah, ok, that was just a quick and dirty trick.  It can still work - if you chuck the startup into the background and run the ps immediately, you should get something:

./start-confuence.sh & ; sleep 1 ; ps -ef | grep -i java

will run the script, give it 1 second to get started and then do the search (I'd expect a crashing Confluence to go for a few seconds before it fails)

[jblanco]

Restore a database backup and the confluence pre-update installation path has been fast. I couldn't spend the time needed for troubleshooting now. I think will skip this upgrade for now.

[jblanco]

-- duplicated --