Issue with upgrading confluence to latest version

[Fluccs Australia]

Hello Support,

We have tried to upgrade directly to the latest version of Confluence 8.

and got the error

~~~

confluence + Database: The database collation 'utf8_bin' is not supported by Confluence. You need to use 'utf8mb4_bin'.

So updated the database charset and collation and then tried to upgrade confluence but got the error:

Exception in thread "I/O dispatcher 2" Exception in thread "I/O dispatcher 1" Exception in thread "I/O dispatcher 5" java.lang.OutOfMemoryError: Java heap space

Exception in thread "Navlink Plugin Executor:thread-1" java.lang.IllegalMonitorStateException

        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(Unknown Source)

        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)

        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)

        at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

        at java.base/java.lang.Thread.run(Unknown Source)

java.lang.OutOfMemoryError: Java heap space

Exception in thread "I/O dispatcher 6" java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

Exception in thread "I/O dispatcher 8" java.lang.OutOfMemoryError: Java heap space

        at java.base/java.util.Collections$UnmodifiableCollection.iterator(Unknown Source)

        at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:210)

        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)

        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)

        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)

        at java.base/java.lang.Thread.run(Unknown Source)

~~~

Could you please have a check into this and let us know the work around for this.

[Nic Brough -Adaptavist-]

Welcome to the Atlassian Community!

It would be useful to know what version of Confluence you started with.

But the error message is pretty clear, the system has not got enough memory to do the work.  Confluence's functionality and capabilities have grown over the years, and that means that more recent versions need more memory space to work in.

What are the -Xms and -Xmx settings you have for this server? 

And (very roughly) how big is the system?  A few thousand pages, tens of thousands, hundreds of thousands? More?

[Fluccs Australia]

Hi @Nic Brough -Adaptavist- 

Thanks for getting back to us

We had an 8GB RAM server and we increased the RAM to 16GB to find out the same problem.

The  -Xms and -Xmx settings in bin/setenv.sh are currently set to 
CATALINA_OPTS="-Xms1024m -Xmx1024m ${CATALINA_OPTS}"

However, we did try to increase it to 2GB... should this be increased prior we initiate the upgrade?

There are hundreds of pages yes.

Regards

Andy

[Nic Brough -Adaptavist-]

The short answer is "yes". 

Your data sizes are not a problem, your server looks like it is more than capable, so I am driven towards the memory allocations.

For smaller and older Confluences, -Xmx1024m is not a bad setting, but as you have increased the server memory so much, you should take advantage of it.    Today, I would give Confluence 4Gb to work with as the starting point.

[Fluccs Australia]

did as you suggested, and now when we upgrade from 7.18.1 to 7.20

we get an error below

executing using dedicated user
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/atlassian/confluence/jre//bin/java
2023-01-26 15:11:07,489 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /opt/atlassian/confluence
Using CATALINA_HOME: /opt/atlassian/confluence
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Using JRE_HOME: /opt/atlassian/confluence/jre/
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8000] (base port [8000] and offset [0]). Tomcat may not be running.
Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Error stopping Catalina
java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/java.net.Socket.<init>(Unknown Source)
at java.base/java.net.Socket.<init>(Unknown Source)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:667)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:391)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:481)

The stop command failed. Attempting to signal the process to stop through OS signal.
Tomcat stopped.

--------------------

catalina.out error:

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-15-26.log instead.
26-Jan-2023 15:15:27.638 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
26-Jan-2023 15:15:27.671 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
26-Jan-2023 15:15:27.672 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8090]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:275)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:230)
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1227)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1240)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:603)
at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1048)

Please help us fix this problem. We have custom entries in our server.xml file 

<Connector port="8090" maxThreads="550"
minSpareThreads="35"
connectionTimeout="20000"

enableLookups="false"
maxHttpHeaderSize="8192"
protocol="HTTP/1.1"
useBodyEncodingForURI="true"
redirectPort="8443"
acceptCount="150"
disableUploadTimeout="true"
scheme="https"
proxyName="confluence.xxxxxxxxxxx.net"
proxyPort="443" />

[root@confluence conf]# netstat -nl | grep 8090
tcp6 0 0 :::8090 :::* LISTEN
[root@confluence conf]#

Regards

Andy

[Fluccs Australia]

adding one more error I found when testing the new configuration

[root@confluence bin]# ./configtest.sh
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/atlassian/confluence/jre//bin/java
2023-01-26 15:34:45,215 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /opt/atlassian/confluence
Using CATALINA_HOME: /opt/atlassian/confluence
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Using JRE_HOME: /opt/atlassian/confluence/jre/
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Jan 26, 2023 3:34:45 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [${confluence.context.path}/synchrony-proxy] does not meet these criteria and has been changed to [/${confluence.context.path}/synchrony-proxy]
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8090"]
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8090"]
Jan 26, 2023 3:34:46 PM org.apache.catalina.util.LifecycleBase handleSubClassException
SEVERE: Failed to initialize component [Connector[HTTP/1.1-8090]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:483)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)

- Andy

[Nic Brough -Adaptavist-]

Thanks for the bold highlights, it makes it a lot easier to see where you think the problem is without shortening the rest of the data that we might want to ask you later!

The last one jumps out to me immediately.  It might be the cause of the other two (I would bet that it is, but I can not be certain).

What it tells us is that there is something else on the server that is running on that port. 

When it comes to Atlassian installation problems, this usually means that while the upgrade has worked (or at least got to a point where it's going to start serving web pages), the old version was still running, and hence hogging the port.  It might not be obvious that the old one was not shut down in full, and sometimes, you can even get "port is in use already" with a completely dead system.

I see you're using a sensible operating system, so there's a few simple commands you can look at to see what is going on.

Visit Confluence in a browser

You may just get a pile of errors, but the important bit is that the connection on that port responds.  If you don't get a "nothing running here" type error, you have something running that you need to find and stop.

• ps -ef
  • Is the most simple one.  A list of processes running - is there any reference to any Confluences there?  If so, try shutting them down (resort to kill -9 if you have to), and restarting the new install.  This is however very very blunt and not the best idea on a production system, it can lead to a bit of data corruption, especially if people are editing pages when you do it.
• netstat -plant  / netstat -plank
  • The flags vary a bit on my boxes, but the results should list all processes with their listening ports whatever your OS.  So you can target the "kill -9" better.
  • But.  Netstat is old, and falling out of use, you may not have it.  Don't bother installing it if you've got the next option.
• ss -tupln
  • is the more modern version which will list active ports

In all cases, adding "| grep 8090" is going to make the output really simple!

Note - in some cases, you need to jump through a lot of hoops to "unlock" a port.  Generally, it is far easier to stop the process doing it from starting on reboot, and then reboot the server!

TLDR: find what is blocking the port and kill it.