Application unresponsive, BLOCKED Threads

[Ram Lakshmanan]

Application became unresponsive, because few hundreds of threads were getting in to the BLOCKED state due to atlassian java SDK. Please refer to the attached thread dump, which was captured from the unresponsive application: 

https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMzAvLS1KSVJBLWR1bXBzLTIwMjEtMDktMzBfMTY0NzAxLnRhci5nei0tMTMtNTAtMzI= 

if you notice starting from dump 4, 5 and 6, number of BLOCKED threads count start to grow rapidly. In Dump #4, there were 104 BLOCKED threads, in Dump #5 there were 189 BLOCKED threads, in Dump #189 there are 237 BLOCKED threads

 All of these threads are BLOCKED because of atlassian SDK. It looks like atlassian SDK is invoking 'java.lang.System.getProperty()' method to read the system properties on every call (instead of caching them).

'java.lang.System.getProperty()' underlyingly uses 'java.util.Hashtable.get()' method. This method is a synchronized method. It means only one thread can invoke this method at a time. If any other threads tries to invoke this method they will be put in to BLOCKED state. Unfortunately atlassian JIRA SDK is invoking this '

java.lang.System.getProperty()' every single time. Due to which application is becoming unresponsive.

Here is the transitive dependency graph indicating the threads that are getting BLOCKED

 Here with I am attaching the stacktrace of few atlassian JIRA code who is invoking this '

java.lang.System.getProperty()' repeatedly.

ForkJoinPool-777-worker-2
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.util.Hashtable.get(Hashtable.java:362)
- locked <0x0000000080f5e118> (a java.util.Properties)
at java.util.Properties.getProperty(Properties.java:969)
at java.util.Properties.getProperty(Properties.java:988)
at java.lang.System.getProperty(System.java:756)
at net.java.ao.atlassian.ConverterUtils.enforceLength(ConverterUtils.java:16)
at net.java.ao.atlassian.ConverterUtils.checkLength(ConverterUtils.java:9)
at net.java.ao.atlassian.AtlassianFieldNameConverter.getName(AtlassianFieldNameConverter.java:46)
at net.java.ao.Common.getValueFieldsNames(Common.java:421)
at net.java.ao.EntityManager.find(EntityManager.java:649)
at net.java.ao.EntityManager.find(EntityManager.java:621)
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:140)
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.find(TenantAwareActiveObjects.java:296)

Camel Thread #6 - backboneThreadPool
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.util.Hashtable.get(Hashtable.java:362)
- locked <0x0000000080f5e118> (a java.util.Properties)
at java.util.Properties.getProperty(Properties.java:969)
at java.util.Properties.getProperty(Properties.java:988)
at java.lang.System.getProperty(System.java:756)
at net.java.ao.atlassian.ConverterUtils.enforceLength(ConverterUtils.java:16)
at net.java.ao.atlassian.ConverterUtils.checkLength(ConverterUtils.java:9)
at net.java.ao.atlassian.AtlassianFieldNameConverter.getName(AtlassianFieldNameConverter.java:46)
at net.java.ao.Common.getPrimaryKeyField(Common.java:257)

http-nio-8080-exec-105 url: /browse/QG-12191,

Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.util.Hashtable.get(Hashtable.java:362)
- locked <0x0000000080f5e118> (a java.util.Properties)
at java.util.Properties.getProperty(Properties.java:969)
at java.util.Properties.getProperty(Properties.java:988)
at java.lang.System.getProperty(System.java:756)
at org.ofbiz.core.entity.EntityFindOptions.<init>(EntityFindOptions.java:124)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:221)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.asList(SelectQueryImpl.java:164)
at com.atlassian.jira.issue.link.DefaultIssueLinkManager.getLinks(DefaultIssueLinkManager.java:343)
at com.atlassian.jira.issue.link.DefaultIssueLinkManager.lambda$new$0(DefaultIssueLinkManager.java:80)
at com.atlassian.jira.issue.link.DefaultIssueLinkManager$$Lambda$338/1649014422.load(Unknown Source)
at com.atlassian.jira.cache.request.RequestCacheImpl.get(RequestCacheImpl.java:58)

Any suggestions on how to resolve this issue?