Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Confluence Server: StuckThread after Upgrade to 7.0.1

We run Confluence Server on 2012 R2 and it was running smoothly until we upgraded from 6.15.8 to 7.0.1.

After the upgrade we encountered the following issues:

Resolved: The top bar was failing to load and in it's place was code. We followed an article about reloading plugins and it resolved the issue for us.

https://confluence.atlassian.com/confkb/how-to-reset-all-confluence-plugins-back-to-their-default-state-through-the-database-859509873.html

Unresolved: when Confluence was started back up and I was at the login screen, everything seemed to be running smoothly. I could browser our public articles and there would be no irregular delays loading the page. However, after I logged in, I noticed significant delay in navigating from page to page (over a minute each time).

I started looking around in the logs and found a few different logs but I was unable to ascertain what the connection between the logs are or what underlying issue is causing the delays.

Here are some of the logs:

From {service-name}-stderr.{today's-date}.log in confluence\logs\

 

org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [https-jsse-nio2-8443-exec-1] (id=[15]) has been active for [62,062] milliseconds (since [9/30/19 8:46 AM]) to serve the same request for [{base-url}] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

 

org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [https-jsse-nio2-8443-exec-1] (id=[15]) was previously reported to be stuck but has completed. It was active for approximately [360,832] milliseconds.

 

From Confluence\Application Data\Confluence\Logs\atlassian-confluence.log:

ERROR [https-jsse-nio2-8443-exec-9] [renderer.internal.http.HttpClientFetcher] fetch Unable to perform a request to: {base-url}/rest/gadgets/1.0/g/messagebundle/en_US/gadget.common%2Cgadget.confluence
-- referer: {base-URL}/display/KB/{article-name} | url: /plugins/macrobrowser/browse-macros.action | traceId: ---- | userName: ----- | action: browse-macros
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[oauth.serviceprovider.internal.AuthenticatorImpl] logOAuthProblem Problem encountered authenticating OAuth client for url "{base-URL}/rest/knowledge-base/1.0/search/KB", error was "consumer_key_unknown", with parameters "{oauth_problem=consumer_key_unknown}"

 

I'm unsure which of these error logs is most significant and I'm not sure where to go from here. I appreciate advice you have to offer!

2 answers

I get those errors also and I am on Confluence 7.0.1 (upgraded from 6.x) and below is my env info. Not sure if you are running the same minus the Linux OS in VMware Fusion.

 

Have you tried running a diagnostic and look in the /var/atlassian/application-data/confluence/logs/atlassian-diagnostics.log file?

 

grep -i stuck /var/atlassian/application-data/confluence/logs/atlassian-confluence.log -c
11 counts

uname -a
Linux localhost.localdomain 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

rpm -qa | grep mysql

mysql-community-common-5.7.27-1.el7.x86_64
mysql-community-client-5.7.27-1.el7.x86_64
mysql-community-libs-compat-5.7.27-1.el7.x86_64
mysql-community-libs-5.7.27-1.el7.x86_64
mysql-community-server-5.7.27-1.el7.x86_64

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"


cat /proc/cpuinfo

rocessor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 94
model name : Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
stepping : 3
microcode : 0xcc
cpu MHz : 2591.775
cache size : 6144 KB

cat /proc/meminfo

MemTotal: 1863236 kB
MemFree: 68260 kB
MemAvailable: 22876 kB
Buffers: 1496 kB
Cached: 28068 kB
SwapCached: 539416 kB
Active: 1236028 kB
Inactive: 426012 kB
Active(anon): 1220520 kB
Inactive(anon): 412600 kB
Active(file): 15508 kB
Inactive(file): 13412 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3905532 kB
SwapFree: 2399740 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 1606888 kB
Mapped: 14456 kB
Shmem: 660 kB
Slab: 67904 kB
SReclaimable: 29868 kB
SUnreclaim: 38036 kB
KernelStack: 8368 kB
PageTables: 9892 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4837148 kB
Committed_AS: 3218280 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 179620 kB
VmallocChunk: 34359310332 kB
HardwareCorrupted: 0 kB
AnonHugePages: 270336 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 75648 kB
DirectMap2M: 2021376 kB
DirectMap1G: 0 kB

Hi Parker,

Thank you for your response. I haven't ran the diagnostics, I'm not sure how and it is very difficult to navigate to anything inside the web-service right now because it won't load half the time and if it does it may take multiple minutes.

I tried to go to the troubleshooting tools section and there was a popup on the right side of the screen that said:

"Your URL doesn't match

Confluence's base URL is set to "{base-url}:{port}" but you are accessing Confluence from "{base-url}:{port}undefined"

 

It's the exact same url with the word "undefined" at the end. I don't know if this is relevant but I thought I would share it anyways.

 

In Troubleshooting and Support Tools it shows the Instance health checks and everything has a passing checkmark. I will try to find the diagnostics you mentioned running.

I was reading some more and I found the database latency for me is very high.

 

Under System Information it says: Database Latency: 48ms

 

I read this should be below 2ms so I'm looking for ways to troubleshoot this.

Follow up: The DB Latency metric usually appears low 0-10 but I occasionally find it higher like when I found it at 48ms.

 

I also found the following two entries in the logs so I'm wondering if this could be related:

 

02-Oct-2019 10:28:16.515 WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

 

02-Oct-2019 10:28:16.530 WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->TOKEN
HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

 

02-Oct-2019 10:32:23.197 SEVERE [https-jsse-nio2-8443-exec-6] org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor.failed Socket accept failed
java.io.IOException: The specified network name is no longer available.

 

I'm wondering if these offer any insight about the issue I'm facing...
I ran a thread dump but I'm not sure how to interpret the results or what to look for.

Thank you for any help!

What database and version are you running? What JDBC driver do you have running?

 

Is the database running on the same server as Confluence?

Microsoft SQL Server

12.00.5687

Driver version: 6.3.0.0

 

The DB is running on a different server than Confluence.

Update on this: we rolled the server / DB back to a 6.x version that was working for us. I'm still searching for advice because we want to be able to upgrade, but we don't know what was causing the stuck thread.

 

Thank you for anyone who offers advice on this.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Posted in Confluence

What do you think is the most *delightful* Confluence feature? Comment for a prize!

- Create your own custom emoji 🔥 - "Shake for Feedback" on mobile 📱 - An endless supply of GIFs via GIPHY 🤩 Is there anything quite as nice as a pleasant surprise? Comment below with what...

415 views 23 8
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you