Confluence runs out of memory 1-2 times per week. Is there a better option than nightly restart?

[Bryan Stephenson]

Hello,

I am running "Confluence 4.3.7 - Standalone distribution" on openSUSE Leap 42.2.  It appears to leak memory.  Looking at the VIRT column of the two outputs shown below from the "top" command, the memory has grown from 9.808g on Friday to 9.981g today, three days later.  It has been running a total of four days since the last crash and restart.  The java process is confluence in both cases.  I have a cron job that restarts it once a week which previously was soon enough to avoid it running out of memory and crashing.  But now this has happened twice in one week.  I want to avoid it going down when I am asleep and failing to provide service to my coworkers in Europe until I wake up to restart it.  Is my best option to restart it three times a week (or maybe every night) or would increasing the Java heap size be an option?  Or should I upgrade to a newer version that maybe does not leak memory?  Also, this seems like an awful lot of memory for it to be using.  I don't believe we are doing anything unusual or strenuous with this instance and there should only be around 100 users, most of whom are not accessing the Confluence Wiki most of the time.

I am a new admin for Confluence so any advice is appreciated.


2017-10-13 top output:
----------------------

KiB Mem:   4045208 total,  3922252 used,   122956 free,        8 buffers
KiB Swap:  2103292 total,  1093008 used,  1010284 free.   626232 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND   
 1815 mysql     20   0 5859980 314296      0 S 0.332 7.770 136:18.75 mysqld    
 7584 root      20   0  9.808g 2.755g   7600 S 0.332 71.41  24:23.52 java      
    1 root      20   0  123888   7480   2020 S 0.000 0.185   2:47.21 systemd   
    2 root      20   0       0      0      0 S 0.000 0.000   0:00.73 kthreadd  


2017-10-16 top output:
----------------------

KiB Mem:   4045208 total,  3923984 used,   121224 free,        8 buffers
KiB Swap:  2103292 total,   963492 used,  1139800 free.   694372 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND   
 1767 root      20   0  9.981g 2.721g  11596 S 1.329 70.54  20:31.58 java      
 1815 mysql     20   0 5859980 305032      0 S 0.332 7.541 144:17.28 mysqld    

 
When this crash happens I see the following in /var/log/messages:
 
2017-10-12T00:50:37.358126-07:00 <machine name> kernel: [4185923.786905] Out of memory
: Kill process 9122 (java) score 741 or sacrifice child
2017-10-12T00:50:37.358126-07:00 <machine name> kernel: [4185923.787010] Killed proces
s 9122 (java) total-vm:10343332kB, anon-rss:3436684kB, file-rss:0kB, shmem-rss:1
2kB
2017-10-12T00:50:37.711803-07:00 <machine name> systemd[1]: confluence.service: Main p
rocess exited, code=killed, status=9/KILL
2017-10-12T00:50:37.712208-07:00 <machine name> stop-confluence.sh[4448]: executing as
 current user

 
 
 Thank you,
 
 Bryan
 
 -----
 Bryan Stephenson
 bryan.stephenson@suse.com
 

[AnnWorley]

Hi Bryan,

Confluence is up to version 6.4.2 so it would be a good idea to upgrade to a supported version.

In the meantime, it sounds like the VM may be killing the Confluence process. Please check out this article and see if it fits your issue: Confluence process dies unexpectedly due to Linux OOM-Killer

Thanks,

Ann

[Peter DeWitt]

@Bryan Stephenson, Confluence seems to have this problem from time to time.  Upgrading is the way to go.  It usually only takes the next point update but since you are so far behind upgrade to current like @AnnWorley suggested.

[Bryan Stephenson]

Thank you.  I will update to 6.4.2.