Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Confluence stopps working after setup and first restart due to memory issues

Hi!

I did install confluence server using atlassian-confluence-7.10.2-x64.bin on a Linux Debian Buster with postgresql 10 data base.

I performed the setup without any issues including connecting to the data base, creating an admin user and set up a test space, then I was able to use confluence. But after stopping and starting confluence via /etc/init.d/confluence1 start it doesnt come up again. After some minutes high cpu usage and using >2gb RAM confluence is stopping by exception.

The logs show "java.lang.OutOfMemoryError", some experiments with variables (eg. -Xms300m -Xmx300m and -XX:ReservedCodeCacheSize=2048m) did not help.

Why the hell confluence stops working after instaling???

The machine has 16GB RAM and 300GB SSD storage, with debian linux 10

 

 

3 answers

1 accepted

0 votes
Answer accepted

Hi @Jens S_ 

Although the JVM is throwing an Out of Memory error, it seems it is not complaining about the heap space, so changing Xmx and Xms values may not be helpful in this case.

Sometimes the JVM can throw this error when it is unable to create new threads if the OS user has reached the maximum number of processes.

Would you mind login in the OS terminal as the user running the Confluence processes and sharing the output of the following command?

ulimit -a

 

If you are running Confluence as a service in Linux, it might be the case where the service is constrained to a certain number of processes/tasks.

Take a look at this SuSE KB article on changing the maximum number of tasks/threads/processes allowed by a service.

Let us know if this helps.

Kind regards,
Thiago

for the confluence user the output is the following:

$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 65536
process 62987
nofiles 1024
vmemory(kbytes) unlimited
locks unlimited
rtprio 0

Can you send the Confluence service definition as well?

Hi @Thiago Masutti , what do you mean by this exactly ?

If you are using init.d, there will be a file /etc/init.d/<service name>.

If you are using systemd, there will a .conf file within the /etc/systemd/system/<service name>.service.d/ folder.

 

Either of these two will be your service definition.

#!/bin/bash

### BEGIN INIT INFO
# Provides: confluence
# Required-Start: $remote_fs $syslog $time $named
# Required-Stop: $remote_fs $syslog $time $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Atlassian Confluence
# Description: Atlassian Confluence Server
### END INIT INFO

# Confluence Linux service controller script
cd "/srv/confluence/bin"

case "$1" in
start)
./start-confluence.sh
;;
stop)
./stop-confluence.sh
;;
restart)
./stop-confluence.sh
./start-confluence.sh
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
;;
esac

Thanks.

So your service definition is standard.

The number of processes allowed to the user seems fine.

Now I'm concerned with the number of open files.

As root, edit the /etc/security/limits.conf file and add the following entries:

<os_user_running_confluence> soft nofile 32768
<os_user_running_confluence> hard nofile 32768

 

Try to start the Confluence service and let us know if the same error occurs.

Regards,
Thiago

SOLVED!   Thank you!

For resolution I performed those steps - maybe one of the solved the problem already:

  • confluence soft nofile 32000
    confluence hard nofile 64000
  • correction of the postgresDb (dropped and recreated)
    • was: Collate C and ctype C (due to wrong suggestion from JIRA installer doc)
    • new: all UTF8
  • vim /etc/systemd/system.conf
    [Manager]
    DefaultTasksMax=15288
  • reboot

btw. I did reset the OPTIONS to default values: 

 /srv/confluence/jre//bin/java -Djava.util.logging.config.file=/srv/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options= -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx1024m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/srv/confluence/logs/gc-2021-01-22_00-49-37.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/srv/confluence/logs/gc-2021-01-22_00-49-37.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions -Dignore.endorsed.dirs= -classpath /srv/confluence/bin/bootstrap.jar:/srv/confluence/bin/tomcat-juli.jar -Dcatalina.base=/srv/confluence -Dcatalina.home=/srv/confluence -Djava.io.tmpdir=/srv/confluence/temp org.apache.catalina.startup.Bootstrap start

Consulted docs:

Like Thiago Masutti likes this
0 votes

Hi @Jens S_ ,

300mb memory does not cut it in Confluence. Try these variables

-Xms8000m -Xmx8000m

And update this variable XX:ReservedCodeCacheSize to the default value whatever it is.

thank you.. unfortunately those parameters doesnt help.. I still see such log messages after starting:

[93,640s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.Exception in thread "Navlink Plugin Executor:thread-2" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.base/java.lang.Thread.start0(Native Method) at java.base/java.lang.Thread.start(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)

brbojorque Community Leader Jan 21, 2021

Hi @Jens S_ ,
May I know the full path of the file you've updated the JVM values with?

Check also the available memory by typing `free -h`

Like Jens S_ likes this

Hi, the file is: /srv/confluence/bin/setenv.sh.  

Changes in this file are effectively, e.g.  trying to set the "JVM_MAXIMUM_MEMORY" variable (which can be set in JIRA) leads to error messages in the log.

Starting confluence with used 700MB RAM (of 16GB max.) on the machine ends with used 3,5 GB used after 1,5 minutes .. at this time lot of error messages like this start appearing:

[95,394s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

Also this message is displayd:

Exception in thread "Catalina-utility-2" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

free -h says:

total used free shared buff/cache available
Mem: 16Gi 3,5Gi 10Gi 69Mi 2,2Gi 12Gi

0 votes

Where are you setting the memory variables?  Actually, only the "max heap memory" one matters much here - the others you've mentioned have performance impacts and could throw other errors, but they won't lead to "out of memory".

-Xmx300m is the right format (I mention this because some people seem to think it is case insensitive - it's not), and although 300m is too low for Confluence, your next test with 8000m should have worked, which makes me suspect that your numbers might simply not be being applied.

When you get the error, could you look back up through the log to the point where it started?  Confluence echoes out the memory settings it has ended up running with and my guess is that it's not inherited the one you have set for some reason.

Hi, as far I know Xms is minimum and Xmx maximum heap size. Thats why I tested it with this set parameters - unfortunately without a solution :

Using CATALINA_OPTS: -XX:ReservedCodeCacheSize=2000m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options='' -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx8000m -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/srv/confluence/logs/gc-2021-01-22_11-44-25.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/srv/confluence/logs/gc-2021-01-22_11-44-25.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions

Could it be that there are other java / memory parameters needs to be changed on the machine?

What does the application log say the parameters are coming out as when you start the applicaiton?

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
TAGS
Community showcase
Published in Confluence

Announcing Team Calendars in Confluence Data Center

Hi Community! We're thrilled to share that Team Calendars for Confluence is now a built-in feature for Confluence Data Center releases 7.11 and beyond.  A long time favorite,  Team Cale...

177 views 0 6
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you