We are seeing very poor performance of our JIRA report macros. Specificaly, the JIRA Report blueprint-created pages. When we refresh these pages, it takes a very long time (30-60 seconds) to load. While loading, we observe 4 "JIRA macros executor" threads running. In fact, these threads have been "running" longer than they have been "waiting". Sometimes these pages simply timeout.
Our production JIRA server is not exceptionally busy AND the same filters that the Confluence report page uses return very quickly in JIRA.
Any ideas what could be causing such poor performance in the Confluence pages that access JIRA content?
Here is a thread dump:
2014-11-25 09:25:32
"Jira macros executor:thread-1" - Thread t@395
java.lang.Thread.State: RUNNABLE
at java.math.BigInteger.mulAdd(BigInteger.java:1935)
at java.math.BigInteger.montReduce(BigInteger.java:1880)
at java.math.BigInteger.oddModPow(BigInteger.java:1850)
at java.math.BigInteger.modPow(BigInteger.java:1599)
at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)
at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)
at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2087)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
- locked <a89a00a> (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3970)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)
at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)
at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXML(DefaultJiraIssuesManager.java:90)
at com.atlassian.confluence.extra.jira.CacheJiraIssuesManager.retrieveXML(CacheJiraIssuesManager.java:41)
at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXMLAsChannel(DefaultJiraIssuesManager.java:172)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.populateContextMapForStaticCountIssues(JiraIssuesMacro.java:907)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.createContextMapFromParams(JiraIssuesMacro.java:495)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.execute(JiraIssuesMacro.java:1195)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <5189a5d5> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Jira macros executor:thread-2" - Thread t@396
java.lang.Thread.State: RUNNABLE
at java.math.BigInteger.mulAdd(BigInteger.java:1930)
at java.math.BigInteger.squareToLen(BigInteger.java:1319)
at java.math.BigInteger.oddModPow(BigInteger.java:1849)
at java.math.BigInteger.modPow(BigInteger.java:1599)
at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)
at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)
at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2087)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
- locked <14b76357> (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3970)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)
at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)
at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXML(DefaultJiraIssuesManager.java:90)
at com.atlassian.confluence.extra.jira.CacheJiraIssuesManager.retrieveXML(CacheJiraIssuesManager.java:41)
at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXMLAsChannel(DefaultJiraIssuesManager.java:172)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.populateContextMapForStaticCountIssues(JiraIssuesMacro.java:907)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.createContextMapFromParams(JiraIssuesMacro.java:495)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.execute(JiraIssuesMacro.java:1195)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <79868617> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Jira macros executor:thread-3" - Thread t@397
java.lang.Thread.State: RUNNABLE
at java.math.BigInteger.mulAdd(BigInteger.java:1930)
at java.math.BigInteger.montReduce(BigInteger.java:1880)
at java.math.BigInteger.oddModPow(BigInteger.java:1850)
at java.math.BigInteger.modPow(BigInteger.java:1599)
at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)
at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)
at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2087)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
- locked <24bb575c> (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3970)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)
at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)
at com.atlassian.confluence.plugins.jiracharts.Base64JiraChartImageService.getBase64JiraChartImageModel(Base64JiraChartImageService.java:39)
at com.atlassian.confluence.plugins.jiracharts.render.JiraImageChart.getImageSourceModel(JiraImageChart.java:53)
at com.atlassian.confluence.plugins.jiracharts.render.PieChart.setupContext(PieChart.java:61)
at com.atlassian.confluence.plugins.jiracharts.JiraChartMacro.execute(JiraChartMacro.java:93)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <5535955e> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Jira macros executor:thread-4" - Thread t@420
java.lang.Thread.State: RUNNABLE
at java.math.BigInteger.mulAdd(BigInteger.java:1930)
at java.math.BigInteger.squareToLen(BigInteger.java:1319)
at java.math.BigInteger.oddModPow(BigInteger.java:1849)
at java.math.BigInteger.modPow(BigInteger.java:1599)
at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)
at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)
at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2087)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)
at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
- locked <51e4faa7> (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3970)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)
at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)
at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)
at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)
at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXML(DefaultJiraIssuesManager.java:90)
at com.atlassian.confluence.extra.jira.CacheJiraIssuesManager.retrieveXML(CacheJiraIssuesManager.java:41)
at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXMLAsChannel(DefaultJiraIssuesManager.java:172)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.populateContextMapForStaticCountIssues(JiraIssuesMacro.java:907)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.createContextMapFromParams(JiraIssuesMacro.java:495)
at com.atlassian.confluence.extra.jira.JiraIssuesMacro.execute(JiraIssuesMacro.java:1195)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)
at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <64a41f41> (a java.util.concurrent.ThreadPoolExecutor$Worker)
