unexpeded shutdowns from Jiira and/or Confluence

[Ruediger Holdmann]

Hello,

happy new year to all of you.

We are experiencing some very strange problems using Jira and Confluence. I have to admit although I am the ServerAdmin I do not know very much about these Tools. I am responsible for the Server on which your Tool is running, but basically that's it.  I hope I do not embarrass myself. Ok, let's start:

It is a Ubuntu Server, running 14.04.5 LTS 64 bit as a Virtual Server with most current Virtualbox-Tool.

free -m for the Virtual-Server says:

All     used       free  shared Puffer      Cache
Speicher:      12015      10149       1865          5        218       3477
-/+ Puffer/Cache:       6454       5561
SWAP:       8189          0       8189

The virtual-server has 4 CPUs and 12GB of RAM in total. I've increased this from 8GB because of the problems, therefore the actual size of SWAP and virtual RAM are different. The HDD is 60GB of this VirtualServer, 50% is free space.

Without any visible occasion nor other Issues, Jiira and/or Confluence are shutting down down on their own. It happens randomly. I have to shutdown the leftover first and run then jiira from shell "sudo service jiira start". I have to wait before I can carry on because this process is eating the entire CPUs for about 2 Minutes. Then I can run confluence "sudo service confluence start" and wait again for 3-5 Minutes, because CPUs are again entirely eaten by this process.

If this is done, both Services are running smoothly. But maybe days or weeks later it's happening again.

When I am checking syslog for the Virtualserver I am finding (i.e.)

Jan  2 09:06:54 TCDevProcess kernel: [246390.698922] Out of memory: Kill process 2377 (java) score 86 or sacrifice child
Jan  2 09:06:54 TCDevProcess kernel: [246390.700837] Killed process 2377 (java) total-vm:4784888kB, anon-rss:1748332kB, file-rss:28504kB
Jan  2 09:06:54 TCDevProcess kernel: [246390.820328] swap_free: Bad swap file entry 2000000000000000
Jan  2 09:06:54 TCDevProcess kernel: [246390.820736] BUG: Bad page map in process java  pte:00000020 pmd:30f53d067
Jan  2 09:06:54 TCDevProcess kernel: [246390.821080] addr:00007f4ad99e2000 vm_flags:08000071 anon_vma:          (null) mapping:          (null) index:7f4ad99e2
Jan  2 09:06:54 TCDevProcess kernel: [246390.821803] file:          (null) fault:          (null) mmap:          (null) readpage:          (null)
Jan  2 09:06:54 TCDevProcess kernel: [246390.822649] CPU: 0 PID: 9208 Comm: java Tainted: G    B           4.4.0-78-generic #99~14.04.2-Ubuntu
Jan  2 09:06:54 TCDevProcess kernel: [246390.822655] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jan  2 09:06:54 TCDevProcess kernel: [246390.822661]  0000000000000000 ffff8801ff24ba88 ffffffff813dd70c 00007f4ad99e2000
Jan  2 09:06:54 TCDevProcess kernel: [246390.822669]  00003ffffffff000 ffff8801ff24bad0 ffffffff811b1194 ffffffff811c74fe
Jan  2 09:06:54 TCDevProcess kernel: [246390.822675]  00000007f4ad99e2 ffff88030f53df10 00007f4ad99e2000 0000000000000020
Jan  2 09:06:54 TCDevProcess kernel: [246390.822682] Call Trace:
Jan  2 09:06:54 TCDevProcess kernel: [246390.822695]  [<ffffffff813dd70c>] dump_stack+0x63/0x87
Jan  2 09:06:54 TCDevProcess kernel: [246390.822704]  [<ffffffff811b1194>] print_bad_pte+0x1e4/0x290
Jan  2 09:06:54 TCDevProcess kernel: [246390.822711]  [<ffffffff811c74fe>] ? swap_info_get+0x7e/0xe0
Jan  2 09:06:54 TCDevProcess kernel: [246390.822718]  [<ffffffff811b29db>] unmap_page_range+0x4cb/0x7b0
Jan  2 09:06:54 TCDevProcess kernel: [246390.822724]  [<ffffffff811b2d41>] unmap_single_vma+0x81/0xf0
Jan  2 09:06:54 TCDevProcess kernel: [246390.822730]  [<ffffffff811b3647>] unmap_vmas+0x47/0x90
Jan  2 09:06:54 TCDevProcess kernel: [246390.822737]  [<ffffffff811bc5f8>] exit_mmap+0x98/0x150
Jan  2 09:06:54 TCDevProcess kernel: [246390.822745]  [<ffffffff8107bd07>] mmput+0x57/0x110
Jan  2 09:06:54 TCDevProcess kernel: [246390.822751]  [<ffffffff81081435>] do_exit+0x255/0xae0
Jan  2 09:06:54 TCDevProcess kernel: [246390.822758]  [<ffffffff81081d3f>] do_group_exit+0x3f/0xa0
Jan  2 09:06:54 TCDevProcess kernel: [246390.822765]  [<ffffffff8108d4bc>] get_signal+0x1cc/0x610
Jan  2 09:06:54 TCDevProcess kernel: [246390.822773]  [<ffffffff8102d4a8>] do_signal+0x28/0x6c0
Jan  2 09:06:54 TCDevProcess kernel: [246390.822780]  [<ffffffff810b9bb5>] ? pick_next_task_fair+0x325/0x4b0
Jan  2 09:06:54 TCDevProcess kernel: [246390.822788]  [<ffffffff81807679>] ? __schedule+0x109/0x980
Jan  2 09:06:54 TCDevProcess kernel: [246390.822795]  [<ffffffff8107984c>] exit_to_usermode_loop+0x59/0xa2
Jan  2 09:06:54 TCDevProcess kernel: [246390.822802]  [<ffffffff81003a16>] prepare_exit_to_usermode+0x26/0x30
Jan  2 09:06:54 TCDevProcess kernel: [246390.822809]  [<ffffffff8180c265>] retint_user+0x8/0x13
Jan  2 09:06:56 TCDevProcess kernel: [246392.480120] BUG: Bad rss-counter state mm:ffff88030f144c00 idx:2 val:-1
Jan  2 09:09:01 TCDevProcess CRON[9567]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/


First I believe Tomcat and/or the JavaVM are running out of memory. I've googled around and I got really confused because of the many different opinions about increasing memory for those or not.

To be honest, I don't really know what to do and hope anyone here can assist me.

best regards

R.Holdmann

[Shannon S]

Hi R.Holdmann,

Have you also increased Jira and Confluence's individual memory?

Have a look at the following:

• Confluence: How to fix out of memory errors
• Jira: Increasing Jira Application Memory

Please try that if you haven't already, and in either case, send us the settings you have from the ones mentioned in the articles.

Regards,

Shannon

[Ruediger Holdmann]

Thx Shannon for replying.

I did not increase the Memory already, because of my confusion. But well, if you say so I will do it right now.

[Ruediger Holdmann]

I've increased Minimum from 384m to 512m, and Maximum from 768m to 1024m according to your FAQ. We'll see if this could resolve our problems.  Otherwise I will raise those values again with 256m steps.

But why are Jira und Confluence eating the entire CPUs during startup? It really worries me a little bit because of the HostServer. It's a 24Core, but I am running several VirtualServers on this. What can I do to help your Tools to startup without that huge CPU-Usage?

[Ruediger Holdmann]

Confluence already has 2G. I am not changing more than one value at the time.

CATALINA_OPTS="-Xms2g -Xmx2g -XX:+UseG1GC ${CATALINA_OPTS}"

[Andy Heinzer]

Hi @Ruediger Holdmann

From looking at your initial errors, I think you are running into this problem: How to Resolve JIRA Process being Terminated Unexpectedly in Linux.  This KB describes a scenario where the application (jira or confluence) can actually start using more memory than the system currently has available to it.  As a result the linux kernel will kill the process in order to make sure it can continue to function.

When this happens Jira does not have a chance to log it's shutdown.  Instead you see in the /var/log/messages or /var/log/syslog the kind of out_of_memory message you posted to begin with.

The system resources might not be sufficient to run both Jira and confluence on the same machine at the same time.  I would recommend checking out the Jira Sizing Guide and the Confluence Server Hardware Requirement Guide as one way to understand how much resources are needed for each application based on how you are using each application (number of users, spaces, issues, etc).   Even if you are running smaller instances of each application, both guides are created for running each application on a stand alone server.  So you will likely need to make sure the system has more than enough resources to run both at the same time (ie, stacking the requirements of each application). 

For example, the confluence guide is expecting at least a quad core 2GHz cpu, just for running confluence by itself.  However Jira CPU requirements can scale from a simply dual core, up very quickly to multiple quad cores for enterprise environments.  To run both of these at the same time, you might need to add a few more CPU cores to this VM in order to run both applications smoothly.

In addition to the application requirements are also concerns about other applications and the resources these applications are using on this machine.  Most commonly this can be a problem if you are also running the SQL database for these application on the same server, as SQL will tend to be a rather intense resource using application itself.  And both JIra and confluence depend on SQL being up and ready to read/write content in order for these applications to work.

I would be interested to learn more about the sizes of your Jira and confluence instances, such as number of users in each, number of issues in Jira, and number of spaces in confluence.  With this information perhaps we can make a better recommendation for what hardware resources should exist on the system to run these applications well.

Regards,

Andy

[Ruediger Holdmann]

Please apologize I am going to pick up this old thread again.

After the changes I've made like I've described above those unexpected Shutdowns happened less but they've still happened.

I am going to increase Memory with steps of 256mm maxium twice to see if this is helping or not.