Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,463,508
Community Members
 
Community Events
176
Community Groups

Issue with upgrading confluence to latest version

Hello Support,

We have tried to upgrade directly to the latest version of Confluence 8.

 

and got the error
~~~
confluence + Database: The database collation 'utf8_bin' is not supported by Confluence. You need to use 'utf8mb4_bin'.
So updated the database charset and collation and then tried to upgrade confluence but got the error:
Exception in thread "I/O dispatcher 2" Exception in thread "I/O dispatcher 1" Exception in thread "I/O dispatcher 5" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Navlink Plugin Executor:thread-1" java.lang.IllegalMonitorStateException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 6" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 8" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Collections$UnmodifiableCollection.iterator(Unknown Source)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:210)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Unknown Source)
~~~
Could you please have a check into this and let us know the work around for this.

1 answer

0 votes

Welcome to the Atlassian Community!

It would be useful to know what version of Confluence you started with.

But the error message is pretty clear, the system has not got enough memory to do the work.  Confluence's functionality and capabilities have grown over the years, and that means that more recent versions need more memory space to work in.

What are the -Xms and -Xmx settings you have for this server? 

And (very roughly) how big is the system?  A few thousand pages, tens of thousands, hundreds of thousands? More?

Hi @Nic Brough -Adaptavist- 

 

Thanks for getting back to us

 

We had an 8GB RAM server and we increased the RAM to 16GB to find out the same problem.

 

The  -Xms and -Xmx settings in bin/setenv.sh are currently set to 
CATALINA_OPTS="-Xms1024m -Xmx1024m ${CATALINA_OPTS}"

However, we did try to increase it to 2GB... should this be increased prior we initiate the upgrade?

 

There are hundreds of pages yes.

 

Regards

Andy

The short answer is "yes". 

Your data sizes are not a problem, your server looks like it is more than capable, so I am driven towards the memory allocations.

For smaller and older Confluences, -Xmx1024m is not a bad setting, but as you have increased the server memory so much, you should take advantage of it.    Today, I would give Confluence 4Gb to work with as the starting point.

did as you suggested, and now when we upgrade from 7.18.1 to 7.20

 

we get an error below

 

executing using dedicated user
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/atlassian/confluence/jre//bin/java
2023-01-26 15:11:07,489 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /opt/atlassian/confluence
Using CATALINA_HOME: /opt/atlassian/confluence
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Using JRE_HOME: /opt/atlassian/confluence/jre/
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8000] (base port [8000] and offset [0]). Tomcat may not be running.
Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Error stopping Catalina
java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/java.net.Socket.<init>(Unknown Source)
at java.base/java.net.Socket.<init>(Unknown Source)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:667)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:391)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:481)

The stop command failed. Attempting to signal the process to stop through OS signal.
Tomcat stopped.

 

--------------------

catalina.out error:

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-15-26.log instead.
26-Jan-2023 15:15:27.638 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
26-Jan-2023 15:15:27.671 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
26-Jan-2023 15:15:27.672 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8090]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:275)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:230)
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1227)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1240)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:603)
at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1048)

 

Please help us fix this problem. We have custom entries in our server.xml file 

 

<Connector port="8090" maxThreads="550"
minSpareThreads="35"
connectionTimeout="20000"

enableLookups="false"
maxHttpHeaderSize="8192"
protocol="HTTP/1.1"
useBodyEncodingForURI="true"
redirectPort="8443"
acceptCount="150"
disableUploadTimeout="true"
scheme="https"
proxyName="confluence.xxxxxxxxxxx.net"
proxyPort="443" />

 

[root@confluence conf]# netstat -nl | grep 8090
tcp6 0 0 :::8090 :::* LISTEN
[root@confluence conf]#

 

Regards

Andy

adding one more error I found when testing the new configuration

 

[root@confluence bin]# ./configtest.sh
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/atlassian/confluence/jre//bin/java
2023-01-26 15:34:45,215 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /opt/atlassian/confluence
Using CATALINA_HOME: /opt/atlassian/confluence
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Using JRE_HOME: /opt/atlassian/confluence/jre/
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Jan 26, 2023 3:34:45 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [${confluence.context.path}/synchrony-proxy] does not meet these criteria and has been changed to [/${confluence.context.path}/synchrony-proxy]
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8090"]
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8090"]
Jan 26, 2023 3:34:46 PM org.apache.catalina.util.LifecycleBase handleSubClassException
SEVERE: Failed to initialize component [Connector[HTTP/1.1-8090]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:483)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.Net.bind(Unknown Source)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)

 

- Andy

Thanks for the bold highlights, it makes it a lot easier to see where you think the problem is without shortening the rest of the data that we might want to ask you later!

The last one jumps out to me immediately.  It might be the cause of the other two (I would bet that it is, but I can not be certain).

What it tells us is that there is something else on the server that is running on that port. 

When it comes to Atlassian installation problems, this usually means that while the upgrade has worked (or at least got to a point where it's going to start serving web pages), the old version was still running, and hence hogging the port.  It might not be obvious that the old one was not shut down in full, and sometimes, you can even get "port is in use already" with a completely dead system.

I see you're using a sensible operating system, so there's a few simple commands you can look at to see what is going on.

Visit Confluence in a browser

You may just get a pile of errors, but the important bit is that the connection on that port responds.  If you don't get a "nothing running here" type error, you have something running that you need to find and stop.

  • ps -ef
    • Is the most simple one.  A list of processes running - is there any reference to any Confluences there?  If so, try shutting them down (resort to kill -9 if you have to), and restarting the new install.  This is however very very blunt and not the best idea on a production system, it can lead to a bit of data corruption, especially if people are editing pages when you do it.
  • netstat -plant  / netstat -plank
    • The flags vary a bit on my boxes, but the results should list all processes with their listening ports whatever your OS.  So you can target the "kill -9" better.
    • But.  Netstat is old, and falling out of use, you may not have it.  Don't bother installing it if you've got the next option.
  • ss -tupln
    • is the more modern version which will list active ports

In all cases, adding "| grep 8090" is going to make the output really simple!

Note - in some cases, you need to jump through a lot of hoops to "unlock" a port.  Generally, it is far easier to stop the process doing it from starting on reboot, and then reboot the server!

TLDR: find what is blocking the port and kill it.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
TAGS

Atlassian Community Events