Issue with upgrading confluence to latest version

Question

Hello Support,

We have tried to upgrade directly to the latest version of Confluence 8.

and got the error~~~confluence + Database: The database collation 'utf8_bin' is not supported by Confluence. You need to use 'utf8mb4_bin'.So updated the database charset and collation and then tried to upgrade confluence but got the error:Exception in thread "I/O dispatcher 2" Exception in thread "I/O dispatcher 1" Exception in thread "I/O dispatcher 5" java.lang.OutOfMemoryError: Java heap spaceException in thread "Navlink Plugin Executor:thread-1" java.lang.IllegalMonitorStateException        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(Unknown Source)        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)        at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)        at java.base/java.lang.Thread.run(Unknown Source)java.lang.OutOfMemoryError: Java heap spaceException in thread "I/O dispatcher 6" java.lang.OutOfMemoryError: Java heap spacejava.lang.OutOfMemoryError: Java heap spaceException in thread "I/O dispatcher 8" java.lang.OutOfMemoryError: Java heap space        at java.base/java.util.Collections$UnmodifiableCollection.iterator(Unknown Source)        at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:210)        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)        at java.base/java.lang.Thread.run(Unknown Source)~~~Could you please have a check into this and let us know the work around for this.

Nic Brough -Adaptavist- · Answer

Thanks for the bold highlights, it makes it a lot easier to see where you think the problem is without shortening the rest of the data that we might want to ask you later!

The last one jumps out to me immediately.  It might be the cause of the other two (I would bet that it is, but I can not be certain).

What it tells us is that there is something else on the server that is running on that port.

When it comes to Atlassian installation problems, this usually means that while the upgrade has worked (or at least got to a point where it's going to start serving web pages), the old version was still running, and hence hogging the port.  It might not be obvious that the old one was not shut down in full, and sometimes, you can even get "port is in use already" with a completely dead system.

I see you're using a sensible operating system, so there's a few simple commands you can look at to see what is going on.

Visit Confluence in a browser

You may just get a pile of errors, but the important bit is that the connection on that port responds.  If you don't get a "nothing running here" type error, you have something running that you need to find and stop.

* ps -ef* Is the most simple one.  A list of processes running - is there any reference to any Confluences there?  If so, try shutting them down (resort to kill -9 if you have to), and restarting the new install.  This is however very very blunt and not the best idea on a production system, it can lead to a bit of data corruption, especially if people are editing pages when you do it.

* netstat -plant  / netstat -plank* The flags vary a bit on my boxes, but the results should list all processes with their listening ports whatever your OS.  So you can target the "kill -9" better.
* But.  Netstat is old, and falling out of use, you may not have it.  Don't bother installing it if you've got the next option.

* ss -tupln* is the more modern version which will list active ports

In all cases, adding "| grep 8090" is going to make the output really simple!

Note - in some cases, you need to jump through a lot of hoops to "unlock" a port.  Generally, it is far easier to stop the process doing it from starting on reboot, and then reboot the server!

TLDR: find what is blocking the port and kill it.

Fluccs Australia · Answer

adding one more error I found when testing the new configuration

[root@confluence bin]# ./configtest.sh
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/atlassian/confluence/jre//bin/java
2023-01-26 15:34:45,215 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /opt/atlassian/confluence
Using CATALINA_HOME: /opt/atlassian/confluence
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Using JRE_HOME: /opt/atlassian/confluence/jre/
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Jan 26, 2023 3:34:45 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [${confluence.context.path}/synchrony-proxy] does not meet these criteria and has been changed to [/${confluence.context.path}/synchrony-proxy]
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8090"]
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8090"]
Jan 26, 2023 3:34:46 PM org.apache.catalina.util.LifecycleBase handleSubClassException
SEVERE: Failed to initialize component [Connector[HTTP/1.1-8090]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:483)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)

- Andy

Fluccs Australia · Answer

did as you suggested, and now when we upgrade from 7.18.1 to 7.20 we get an error below executing using dedicated user If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out --------------------------------------------------------------------------- Using Java: /opt/atlassian/confluence/jre//bin/java 2023-01-26 15:11:07,489 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required --------------------------------------------------------------------------- Using CATALINA_BASE: /opt/atlassian/confluence Using CATALINA_HOME: /opt/atlassian/confluence Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp Using JRE_HOME: /opt/atlassian/confluence/jre/ Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer SEVERE: Could not contact [localhost:8000] (base port [8000] and offset [0]). Tomcat may not be running. Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer SEVERE: Error stopping Catalina java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at java.base/java.net.Socket.(Unknown Source) at java.base/java.net.Socket.(Unknown Source) at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:667) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:391) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:481) The stop command failed. Attempting to signal the process to stop through OS signal. Tomcat stopped. -------------------- catalina.out error: NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-15-26.log instead. 26-Jan-2023 15:15:27.638 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"] 26-Jan-2023 15:15:27.671 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"] 26-Jan-2023 15:15:27.672 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8090]] org.apache.catalina.LifecycleException: Protocol handler initialization failed at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.startup.Catalina.load(Catalina.java:724) at org.apache.catalina.startup.Catalina.load(Catalina.java:746) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475) Caused by: java.net.BindException: Address already in use at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Unknown Source) at java.base/sun.nio.ch.Net.bind(Unknown Source) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:275) at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:230) at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1227) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1240) at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:603) at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80) at org.apache.catalina.connector.Connector.initInternal(Connector.java:1048) Please help us fix this problem. We have custom entries in our server.xml file [root@confluence conf]# netstat -nl | grep 8090 tcp6 0 0 :::8090 :::* LISTEN [root@confluence conf]# Regards Andy