Insuffecient memory when running hira, confluence and bit.

[Poietis]

Hello, I have a big problem, I jave installed Jira, Confluence and BitBucket on a VM.

Confluence and Jira keep crashing when they are running together.

I have this message:

2018-04-03 11:06:48,898 INFO [localhost-startStop-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$1 Populating setup configuration if running with Cluster mode...
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6bcd580000, 65536, 1) failed; error='Ne peut allouer de la mémoire' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid25919.log
#
# Compiler replay data is saved as:
# /tmp/replay_pid25919.log

My VM' RAM is 4096Mb, 4 Virtual CPUs (2 virtual sockets, 2 cores per vitu socket.).

[Nic Brough -Adaptavist-]

What have you set their allocated memories to?  Look for the Xmx settings for Jira, Confluence and Synchrony (if you've got collaborative editing enabled).

You need to give all of them a maximum limit for their memory, and allow room for the operating system. 

[Poietis]

Thanks for the (quick) reply!

In /opt/atlassian/jira/bin/setenv.sh:

JVM_MINIMUM_MEMORY="384m"
JVM_MAXIMUM_MEMORY="768m"

I tried
#JVM_MINIMUM_MEMORY="512m"
#JVM_MAXIMUM_MEMORY="1024m"

But it is does not fix the issue. Is there any others options to change?

[Nic Brough -Adaptavist-]

Check the settings for Synchrony and Confluence as well.

[Poietis]

I  don't have Synchrony.

Here is what contains /opt/atlassian/jira/bin/setenv.sh

CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="${START_CONFLUENCE_JAVA_OPTS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dconfluence.context.path=${CONFLUENCE_CONTEXT_PATH} ${CATALINA_OPTS}"

I think  it 's the line

CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}"

that sets the memory options for java, is it correct?

I don't want to create 3 individual VM for Bitbucket, confluence, and jira (I want to  let thoses 3 services run on one VM).
Have you an idea?

If you need additionals logs / config file, I can provide them.

Thank you.

[Nic Brough -Adaptavist-]

Sort of. The memory values vary in location between applications.  It's hard working them all out from scratch, I tend to simply rely on "ps -ef | grep -i java" to see what each one has ended up with after all the startup scripts have run.  Look at the -Xmx settings for every java process running.

When you say you're not running Synchrony, does that mean you have deliberately turned off Collaborative editing in Confluence?  (If not, then you are running it)

[Poietis]

Here is the ouput for ps -ef | grep -i java

About Synchrony, I didn't modify anything in confluence, so I think i'm using it.
(And the last lines of the output of ps shows synchrony.)

About the -Xmx values, is my output seems correct to you?

atlbitb+   918     1  0 mars30 ?       00:11:48 /opt/atlassian/bitbucket/5.9.0/jre/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss320k -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -Dpath.conf=/var/atlassian/application-data/bitbucket/shared/search -Des.path.home=/opt/atlassian/bitbucket/5.9.0/elasticsearch -cp /opt/atlassian/bitbucket/5.9.0/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/atlassian/application-data/bitbucket/log/search/elasticsearch.pid -Epath.conf=/var/atlassian/application-data/bitbucket/shared/search

atlbitb+   937     1  0 mars30 ?       00:34:42 /opt/atlassian/bitbucket/5.9.0/jre/bin/java -classpath /opt/atlassian/bitbucket/5.9.0/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/var/atlassian/application-data/bitbucket -Dbitbucket.install=/opt/atlassian/bitbucket/5.9.0 -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/var/atlassian/application-data/bitbucket/tmp -Djava.library.path=/opt/atlassian/bitbucket/5.9.0/lib/native;/var/atlassian/application-data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start
conflue+ 29384     1 99 13:39 ?        00:04:52 /opt/atlassian/confluence/jre//bin/java -Djava.util.logging.config.file=/opt/atlassian/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dconfluence.context.path= -Datlassian.plugins.startup.options= -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx1024m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/opt/atlassian/confluence/logs/gc-2018-04-03_13-39-25.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -Dignore.endorsed.dirs= -classpath /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/confluence -Dcatalina.home=/opt/atlassian/confluence -Djava.io.tmpdir=/opt/atlassian/confluence/temp org.apache.catalina.startup.Bootstrap start

jira     29606     1 99 13:39 ?        00:03:47 /opt/atlassian/jira/jre//bin/java -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx768m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -classpath /opt/atlassian/jira/bin/bootstrap.jar:/opt/atlassian/jira/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp org.apache.catalina.startup.Bootstrap start

conflue+ 29953 29384 59 13:41 ?        00:00:30 /opt/atlassian/confluence/jre/bin/java -classpath /opt/atlassian/confluence/temp/2.1.0-release-confluence_6.5-1a01ab2d.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/postgresql-42.1.1.jar -Xss2048k -Xmx1g synchrony.core sql
root     30178 25476  0 13:42 pts/0    00:00:00 grep -i java

[Nic Brough -Adaptavist-]

Correct, I think so.  But it is broken.

You've got two Bitbuckets running, which probably isn't going to help, but the main thing here is this sequence of switches:

-Xmx1024m
-Xmx1g
-Xmx1g
-Xmx1g
-Xmx768m

These are the maximum memory allocations for each process.  Try adding them up and comparing the result with your VM's available memory, and don't forget you need to add another row for "operating system space" (Without knowing distro and optimisation, I'd start with allowing 1Gb for a server install.  More if it's a desktop distro, less if it's a minimal or optimised one like DietPi or Arch/Gentoo)