Garbage Collector - Alllocation Failure

[Dirk Hainbach]

Dear support-team,

we face a serious issue with our JIRA application.

2020-03-10T16:44:09.538+0100: 46.686: [GC (Allocation Failure) [PSYoungGen: 625082K->88146K(660992K)] 934837K->397908K(1971712K), 0.0517650 secs] [Times: user=0.16 sys=0.00, real=0.06 secs]

We already have implemented the advices from an earlier thread, but it is not solving our problem.

https://community.atlassian.com/t5/Jira-questions/Java-GC-blocks-Jira/qaq-p/122767

What else do you advice?

[meschg]

Having the same issues since 8.9.0 today I noticed it having issues with the Jira Server <-> Bitbucket Cloud integration.

Issue

• Can not setup the Bitbucket Cloud integration without an error refering to check Jira logs (At first I though it only occurs when using Bitbucket Cloud Integration, but this is probably not true)
• GC Allocation Failures in the logs

• Log-Entries:

  2020-05-28T11:49:06.269+0200: 265,063: [GC (Allocation Failure) [PSYoungGen: 480768K->13701K(587264K)] 1026619K->559560K(1546240K), 0,0495788 secs] [Times: user=0,11 sys=0,00, real=0,04 secs] 
  2020-05-28T11:50:15.493+0200: 334,287: [GC (Allocation Failure) [PSYoungGen: 494469K->11192K(587776K)] 1040328K->557060K(1546752K), 0,0682685 secs] [Times: user=0,11 sys=0,00, real=0,07 secs] 
  2020-05-28T11:51:13.785+0200: 392,579: [GC (Allocation Failure) [PSYoungGen: 499128K->14258K(590336K)] 1044996K->560134K(1549312K), 0,0521206 secs] [Times: user=0,10 sys=0,00, real=0,05 secs] 
  2020-05-28T11:55:27.043+0200: 645,837: [GC (Allocation Failure) [PSYoungGen: 502194K->15155K(589824K)] 1048070K->561039K(1548800K), 0,0802687 secs] [Times: user=0,17 sys=0,00, real=0,08 secs]

Environment

• Debian 10, Linux 4.19.0-9-amd64, Stable, up-to Date
• Server RAM ~2GB Free
• Java Version 1.8.0_202
• Java Vendor AdoptOpenJdk

What I tried so far

• VM Restart of course
• Java VM Memory Statistics
  Total Memory 1820 MB
  Free Memory 901 MB
  Used Memory 919 MB
  Memory Graph
  50% Free (Total: 1820 MB) (Force garbage collection)
• Force Garbage Collection causes no errors

  2020-05-28T11:55:40.062+0200: 658,856: [Full GC (System.gc()) [PSYoungGen: 14757K->0K(512000K)] [ParOldGen: 545891K->550240K(958976K)] 560649K->550240K(1470976K), [Metaspace: 280022K->279990K(1314816K)], 1,9053765 secs] [Times: user=12,44 sys=0,06, real=1,90 secs] 

Question

• Which logging Levels should I change to get more information?
• What should I try?

[Andy Heinzer]

Hi,

Allocation errors usually mean that the Java application in question (Jira) does not have enough memory at the time of this error.  Sometimes just increasing the Xmx value can help, but....

Before you increase the memory Jira has, you might want to make sure that the system memory here in the VM is being dedicated to this system, and not shared between VMs on the host.  In a shared memory vm system could cause the system to be changing the memory that system has available to it on the fly.  That's not ideal for a Java application like Jira that has an expectation to manage its own memory it has to allocate from the operating system it is running upon.

In your current state it looks like Jira's heap is about 2GB, but there is also only about 2GB free of system memory.  We always expect the system to have about 4x the system memory as the heap Jira will use.  That is because it's not just Jira running on that host OS, you have to account for other system processes will also need memory resources here.  I'd suggest having 16GB of system memory for that VM dedicated to it. 

Provided that is true, try setting the -Xmx and the -Xms to the same value such as 4g or 4096m.  This will increase the heap Jira has access to to 4GB and keep it constantly using that heap size.  By setting both of these to be the same, you can prevent Jira from resizing its heap when it is running.  Sometimes Java programs will try to resize to save memory, but these resizing actions managed by the JVM can cause stop-the-world events in which Java applications like Jira will become unresponsive until the resize is complete.  Steps to increase the memory are in Increasing Jira application memory.  As long as you're adjusting the -Xmx, you may as well set the minimum memory value Xms to have the same value. 

Try that, if that doesn't help, let me know how much total system memory is here and perhaps we can look deeper into what is happening here.  But essentially I do believe that a larger heap could help you get past this problem.

Andy

[meschg]

Hi, Andy thanks for my information

Regarding the potential impact

- I solved the Bitbucket Cloud issue with the right configuration not relation there

Regarding the virtual machine

- I am in control over the physical host and I am not using dynamic shared memory.
- The VM was ~11GB before and is now exactly set to 16GiB RAM

The situation in the logs remained, even with higher memory.

/var/atlassian/application-data/jira/log/atlassian-jira.log -> no errors

/opt/atlassian/jira/logs/atlassian-jira-gc-2020-05-29_10-27-40.log.0.current

2020-05-29T10:31:00.950+0200: 199,298: [GC (Allocation Failure) [PSYoungGen: 315972K->22766K(493568K)] 902003K->613472K(1639936K), 0,0820621 secs] [Times: user=0,17 sys=0,01, real=0,08 secs] 
2020-05-29T10:32:50.742+0200: 309,089: [GC (Allocation Failure) [PSYoungGen: 325358K->24474K(497152K)] 916064K->619096K(1643520K), 0,0716646 secs] [Times: user=0,16 sys=0,00, real=0,07 secs]

Solution: Regarding xmx == xms settings

• Looks like this was the right approach
• Initial: JVM Input Arguments
  -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms1024m -Xmx2048m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m
• I now set it too

JVM_MINIMUM_MEMORY="2048m" //From "1024m"
JVM_MAXIMUM_MEMORY="2048m"

• Now with -Xms2048m -Xmx2048m it looks like the error no longer occours
• I did not touch the files since installing 8.9.0. Why is the Jira version shipping with not recommend JVM settings?

By the way: I am also running Confluence on the same VM as Jira
- Confluence Java VM stats
Xmx 1024 MB, Xms 1024 MB, Maximum heap size 1024 MB, Total allocated memory 1024 MB, Used memory 571 MB, Free allocated memory 452 MB
- no GC issues there, looks like really xmx == xms is the issue

kind regards

Gerhard

[Andy Heinzer]

Hi,

Glad to hear it sounds like you got it working better here. 

It's not exactly fair to say that Jira server is shipping with 'not recommended JVM settings'. But I think I understand what you are getting at here. We at Atlassian don't always know how much memory the host system will have when you install Jira. And every environment is different.  So it tends to be a better practice to start with a small default heap, and allocate more memory as needed.  If we start with the same heap and make it too large, many first time users will not even be able to start the application if the system is under-resourced, which is arguably a worse experience for them.

Having a constantly sized heap that is slightly more than Jira will ever actually use will tend to give the best performance.  But we don't really have a great way to auto-detect what is best for each environment and force that into the configuration.  Instead we rely on the admin choosing what is best for their own environment.

I hope that helps to explain this better. Cheers,

Andy

[meschg]

Hi, sorry I did not mean to be unfair or mean. I am happy that we are able to so solve this with your support so fast. Thank you a lot. :)

But actually I personally no longer share the support for absolutely low end system with application with 1GB RAM or even below. I think the Atlassian systems sizing guides needs some adaption to the year 2020. RAM is cheap and the changes of the default heap size in Jira 8.x to 1GB / 2 GB was definitely a step into the right direction. But well lets stop the discussion here. I think we provided good detailed information on the topic and it is easy for to change, if other admins also have this issue.

Kind regards,

[Andy Heinzer]

Hi Dirk,

I understand that you're seeing a number of garbage collector allocation failure error messages in the Jira logs.  This tends to be a good indicator that your Jira application is running out of heap (memory).   So the most simplistic approach is actually to increase the amount of memory Jira has to use in the heap.  Details on how to do that are in Increasing Jira application memory.  The steps can be slightly different between Windows and linux, and again between Windows when running as a service vs running as a user (bat file).

I have no idea how large your Jira instance is, or what the specs of your Jira server are.  But we have a rough guidelines of expected system resources you can view in Jira sizing guide.  For even a small Jira site, I would expect at least 1gb allocated to the Xmx parameter, and the system itself to have at least 8GB of system memory.  As your Jira server size grows, so do the recommended resources to allocate here.

Try this and let me know if you have any questions or concerns about this.

Andy