java.lang.OutOfMemoryError: Java heap space - Confluence

Question

Hello everyone,

I looked through the suggestions, and did not find any solutions.

My Jira Server (Confluence + Jira) crashes every Thursday in the morning.

The catalina log of confluence says:

---

02-Aug-2018 02:04:15.951 SEVERE [SockJS-4] org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler.handleError Unexpected error occurred in scheduled task.
 java.lang.OutOfMemoryError: Java heap space

02-Aug-2018 02:04:15.952 SEVERE [http-nio-8090-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run
 java.lang.OutOfMemoryError: Java heap space

02-Aug-2018 02:04:15.963 SEVERE [http-nio-8090-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint$Poller.run
 java.lang.OutOfMemoryError: Java heap space

---

Installed java version:

java version "1.7.0_65"
OpenJDK Runtime Environment (IcedTea 2.5.1) (7u65-2.5.1-5~deb7u1)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

---

My two Atlassian installations:

/app/atlassian/jira/bin/setenv.sh
/app/atlassian/confluence/bin/setenv.sh

setenv file of jira:

---

#
# One way to set the JIRA HOME path is here via this variable.  Simply uncomment it and set a valid path like /jira/home.  You can of course set it outside in the command terminal.  That will also work.
#
#JIRA_HOME=""

#
#  Occasionally Atlassian Support may recommend that you set some specific JVM arguments.  You can use this variable below to do that.
#
JVM_SUPPORT_RECOMMENDED_ARGS=""

#
# The following 2 settings control the minimum and maximum given to the JIRA Java virtual machine.  In larger JIRA instances, the maximum amount will need to be increased.
#
JVM_MINIMUM_MEMORY="768m"
JVM_MAXIMUM_MEMORY="2048m"

#
# The following are the required arguments for JIRA.
#
JVM_REQUIRED_ARGS='-Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory'

# Uncomment this setting if you want to import data without notifications
#
#DISABLE_NOTIFICATIONS=" -Datlassian.mail.senddisabled=true -Datlassian.mail.fetchdisabled=true -Datlassian.mail.popdisabled=true"

#-----------------------------------------------------------------------------------
#
# In general don't make changes below here
#
#-----------------------------------------------------------------------------------

#-----------------------------------------------------------------------------------
# Prevents the JVM from suppressing stack traces if a given type of exception
# occurs frequently, which could make it harder for support to diagnose a problem.
#-----------------------------------------------------------------------------------
JVM_EXTRA_ARGS="-XX:-OmitStackTraceInFastThrow"

PRGDIR=`dirname "$0"`
cat "${PRGDIR}"/jirabanner.txt

JIRA_HOME_MINUSD=""
if [ "$JIRA_HOME" != "" ]; then
    echo $JIRA_HOME | grep -q " "
    if [ $? -eq 0 ]; then
        echo ""
        echo "--------------------------------------------------------------------------------------------------------------------"
        echo "   WARNING : You cannot have a JIRA_HOME environment variable set with spaces in it.  This variable is being ignored"
        echo "--------------------------------------------------------------------------------------------------------------------"
    else
        JIRA_HOME_MINUSD=-Djira.home=$JIRA_HOME
    fi
fi

JAVA_OPTS="-Xms${JVM_MINIMUM_MEMORY} -Xmx${JVM_MAXIMUM_MEMORY} ${JAVA_OPTS} ${JVM_REQUIRED_ARGS} ${DISABLE_NOTIFICATIONS} ${JVM_SUPPORT_RECOMMENDED_ARGS} ${JVM_EXTRA_ARGS} ${JIRA_HOME_MINUSD} ${START_JIRA_JAVA_OPTS}"

export JAVA_OPTS

# DO NOT remove the following line
JAVA_HOME="/app/atlassian/jira/jre/"; export JAVA_HOME

echo ""
echo "If you encounter issues starting or stopping JIRA, please see the Troubleshooting guide at http://confluence.atlassian.com/display/JIRA/Installation+Troubleshooting+Guide"
echo ""
if [ "$JIRA_HOME_MINUSD" != "" ]; then
    echo "Using JIRA_HOME:       $JIRA_HOME"
fi

# set the location of the pid file
if [ -z "$CATALINA_PID" ] ; then
    if [ -n "$CATALINA_BASE" ] ; then
        CATALINA_PID="$CATALINA_BASE"/work/catalina.pid
    elif [ -n "$CATALINA_HOME" ] ; then
        CATALINA_PID="$CATALINA_HOME"/work/catalina.pid
    fi
fi
export CATALINA_PID

if [ -z "$CATALINA_BASE" ]; then
  if [ -z "$CATALINA_HOME" ]; then
    LOGBASE=$PRGDIR
    LOGTAIL=..
  else
    LOGBASE=$CATALINA_HOME
    LOGTAIL=.
  fi
else
  LOGBASE=$CATALINA_BASE
  LOGTAIL=.
fi

PUSHED_DIR=`pwd`
cd $LOGBASE
cd $LOGTAIL
LOGBASEABS=`pwd`
cd $PUSHED_DIR

echo ""
echo "Server startup logs are located in $LOGBASEABS/logs/catalina.out"

# Set the JVM arguments used to start JIRA. For a description of the options, see
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

#-----------------------------------------------------------------------------------
# This allows us to actually debug GC related issues by correlating timestamps
# with other parts of the application logs.
#-----------------------------------------------------------------------------------
GC_JVM_PARAMETERS=""
GC_JVM_PARAMETERS="-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause ${GC_JVM_PARAMETERS}"
GC_JVM_PARAMETERS="-Xloggc:$LOGBASEABS/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M ${GC_JVM_PARAMETERS}"

CATALINA_OPTS="${GC_JVM_PARAMETERS} ${CATALINA_OPTS}"
export CATALINA_OPTS

---

And setenv file from confluence:

---

# See the CATALINA_OPTS below for tuning the JVM arguments used to start Confluence.

echo "If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide"

# set the location of the pid file
if [ -z "$CATALINA_PID" ] ; then
    if [ -n "$CATALINA_BASE" ] ; then
        CATALINA_PID="$CATALINA_BASE"/work/catalina.pid
    elif [ -n "$CATALINA_HOME" ] ; then
        CATALINA_PID="$CATALINA_HOME"/work/catalina.pid
    fi
fi
export CATALINA_PID

PRGDIR=`dirname "$0"`
if [ -z "$CATALINA_BASE" ]; then
  if [ -z "$CATALINA_HOME" ]; then
    LOGBASE=$PRGDIR
    LOGTAIL=..
  else
    LOGBASE=$CATALINA_HOME
    LOGTAIL=.
  fi
else
  LOGBASE=$CATALINA_BASE
  LOGTAIL=.
fi

PUSHED_DIR=`pwd`
cd $LOGBASE
cd $LOGTAIL
LOGBASEABS=`pwd`
cd $PUSHED_DIR

echo ""
echo "Server startup logs are located in $LOGBASEABS/logs/catalina.out"
# IMPORTANT NOTE: Only set JAVA_HOME or JRE_HOME above this line
# Get standard Java environment variables
if $os400; then
  # -r will Only work on the os400 if the files are:
  # 1. owned by the user
  # 2. owned by the PRIMARY group of the user
  # this will not work if the user belongs in secondary groups
  . "$CATALINA_HOME"/bin/setjre.sh
else
  if [ -r "$CATALINA_HOME"/bin/setjre.sh ]; then
    . "$CATALINA_HOME"/bin/setjre.sh
  else
    echo "Cannot find $CATALINA_HOME/bin/setjre.sh"
    echo "This file is needed to run this program"
    exit 1
  fi
fi

echo "---------------------------------------------------------------------------"
echo "Using Java: $JRE_HOME/bin/java"
CONFLUENCE_CONTEXT_PATH=`$JRE_HOME/bin/java -jar $CATALINA_HOME/bin/confluence-context-path-extractor.jar $CATALINA_HOME`
export CONFLUENCE_CONTEXT_PATH
$JRE_HOME/bin/java -jar $CATALINA_HOME/bin/synchrony-proxy-watchdog.jar $CATALINA_HOME
echo "---------------------------------------------------------------------------"

# Set the JVM arguments used to start Confluence. For a description of the options, see
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="${START_CONFLUENCE_JAVA_OPTS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dconfluence.context.path=${CONFLUENCE_CONTEXT_PATH} ${CATALINA_OPTS}"

export CATALINA_OPTS

---

Thanks in advance for your help!

sreepathy · Answer

The error java.lang.OutOfMemoryError: Java heap space that you are facing actually indicates that your JVM is running out of memory to allocate for objects in the heap. The maximum heap size that you have set in your configuration may be quite low for a Confluence production setup, especially when handling large pages, attachments, or multiple concurrent users. I also suspect that any of the scheduled tasks such as an index rebuild, backup, or a plugin job may have influence on this issue as these tasks may consume a large amount of memory.

So, to tackle this situation, you can slowly start increasing the heap size in the setenv.sh file to 2 GB (-Xmx2048m) or more. This value may change depending on your available system memory. I also feel that your Java version (Java 7) is outdated and no longer supported. Upgrading to Java 8 or a later LTS version can help.

You can analyze the GC logs using tools like GCeasy or yCrash to understand if the JVM is spending excessive time in garbage collection before running out of memory. You should also check if any plugins are consuming high memory. Try to disable or update non-essential plugins and observe the behavior. This has also helped people who were facing such issues. It’s also advisable to review Confluence’s Scheduled Jobs in the admin console to see if a particular task is running around the same time the crash occurs. If you are interested in learning about the  types of OutOfMemoryError, you can check out this blog: Types of OutOfMemoryError, Causes, and Solutions.

Andy Heinzer · Answer

Confluence does not currently support OpenJDK.  It might not be the precise cause of your heap errors, but I can't recommend running this JVM.   Please see the details in Confluence Supported Platforms page.  You should be using a 1.8.xx_xx Oracle JRE or JDK for Confluence and Jira in the current versions (6.x and 7.x respectively).

It is still possible that you would need to increase the Xmx value in your settings.  Check out How to fix out of memory errors by increasing available memory

It provides some additional steps you can take here.

Moses Thomas_08kyb0 · Answer

@Lukas BlauensteinerI think its  time to  increase your memory  on  your instance  please see related  issue here. https://community.atlassian.com/t5/Questions/About-java-heap-space/qaq-p/311318

best!