org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread

[Rhobin Jay Faigones]

Hi,

A lot of these messages are found from catalina logs of confluence. The application was then unaccessible after all apache workers were occupied. The stuck threads are one of the suspected cause of the application crash. However, I want to further understand the issue.

Could somebody me understand the exception trace please? The first 30 lines are:

19-Feb-2025 00:35:44.730 WARNING [Catalina-utility-3] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-396 url: /plugins/pagetree/naturalchildren.action; user: kamendoza] (id=[375224]) has been active for [62,914] milliseconds (since [2/19/25, 12:34 AM]) to serve the same request for [https://confluence.dyson.global.corp/plugins/pagetree/naturalchildren.action?decorator=none&excerpt=false&sort=position&reverse=false&disableLinks=false&expandCurrent=true&placement=sidebar&hasRoot=true&pageId=135432181&treeId=0&startDepth=0&mobile=false&ancestors=236938628&ancestors=297941501&ancestors=156385657&ancestors=285518494&ancestors=244506823&ancestors=135432181&treePageId=328281291&_=1739925276077] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [9] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at java.base@11.0.20.1/java.net.SocketInputStream.socketRead0(Native Method)
at java.base@11.0.20.1/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base@11.0.20.1/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base@11.0.20.1/java.net.SocketInputStream.read(SocketInputStream.java:140)
at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:2023)
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6418)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7579)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:592)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:524)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2979)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:248)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:223)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:446)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.getResultSet(AbstractLoadPlanBasedLoader.java:390)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeQueryStatement(AbstractLoadPlanBasedLoader.java:163)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:104)
at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:285)
at org.hibernate.persister.entity.AbstractEntityPersister.doLoad(AbstractEntityPersister.java:4521)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4511)
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:571)
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:539)
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:208)
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:327)
at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:108)
...

Appreciate any help.

Thanks!

[Radek Dostál]

url: /plugins/pagetree/naturalchildren.action

This doesn't seem like a native endpoint, therefore I think it has something to do with a plugin.

As the stacktrace is cut off I don't see any packages that we could point at, maybe you'd find something non-standard from the full stack that would point us to a plugin key.

From

ancestors=236938628&ancestors=297941501&ancestors=156385657&ancestors=285518494&ancestors=244506823&ancestors=135432181&treePageId=328281291

You could open the tree page id (if it's there), or one of the ancestor pages until you find something to do with "natural children".

Once you've found it, you should see the page's source format so that you can find the macro's key.

Once you have the key, you can go to 'Macro Usage' from global administration, and find which plugin the macro belongs to.

If it is a plugin, check whether you have the right version installed and/or if there are any updates / bugfix releases. In any case, this should be looked at and addressed by the vendor, if their plugin causes crashes and long running threads, it's simply not acceptable for data center standards and should be fixed.

[Fabio Racobaldo _Herzum_]

Hi @Rhobin Jay Faigones ,

stuck threads impact for sure your application and can cause performance issues at application level. You need to perform an investigation at your side in order to identify what processes caused that error. 

In order to perfom that analysis please take a look to the following article https://confluence.atlassian.com/confkb/how-to-analyze-thread-dumps-788039144.html

Moreover, my suggestion is to perform a performance tuning of your application on cpu and jvm parameters.

Hope this helps,

Fabio