nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Jira Software Thread get stucked

Shiroukoji

Hey,

I have actually a Problem with my self-hosted Jira Software. I'm using a fresh install of the docker container atlassian/jira-software:8. The installation works well, but after the installation I can use it for a few minutes and then it freezes. I also can't login anymore, because it freezes.

It tried to access Jira directly or behind an reverse proxy with https, each of them have the same issues.

After taking a look at the logs, every time there are these warning:

[http-nio-8080-exec-11] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://_myURL_/rest/gadget/1.0/login, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

and

[ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8080-exec-10] (id=[29]) has been active for [123,347] milliseconds (since [8/4/19 3:49 PM]) to serve the same request for [https://_myURL_/secure/] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

I set up the Jira JVM to get 4096MB (it uses ~1200MB) memory and it has no limitations for cpu.

Have you any idea how I can fix it?

Greetings

Accepted answers

All comments

Earl McCutcheon

Hello Shiroukoji,

The First warning you noted relating to the "filterFormParameters" is just excess logging and just a false positive, check out the following feature request for some details on the:

https://jira.atlassian.com/browse/JRASERVER-59898

The second warning as well is just warnings, the stuck thread message is to bring our attention to threads that may be causing performance issues, so they help when we are troubleshooting a specific issue. As long as you are not seeing any performance hiccups in tandem with hanging threads they are mostly benign, however in your case you can clearly see the instance is suffering from a freeze up when this is happening so defiantly something performance related. You'll want to look at the logging to see if there are any additional errors that can help narrow this one down.

As a follow up I would recomend that you collect a thread dump while the error is occuring to get additional details on the hanging thread that was stuck, but overall I would recomend focusing on the rest of the logging for additional errors as the first step to get additional context on the failure point, and let me know what you find.

Regards,
Earl

Andy Heinzer

Hi,

I see that you are noticing a stuck thread error in the logs and that this message appears to be related to a time when Jira is unresponsive. I would like to learn more about this unresponsiveness and how you get around this problem in the past. For example, when this happens does Jira eventually recover? Or are you having to stop the process and restart Jira here?

If Jira does not log anything else in the $JIRAHOME/log/atlassian-jira.log or the $JIRAINSTALL/logs/catalina.out, then there are a couple of things that could be happening. Either

The java process running Jira has been terminated. This is known to happen on Linux environments when Jira's memory allocation is set so high that the system runs out of system memory needed for other basic operating system processes, and then the Linux kernel can terminate the process using the most resources. When this happens Jira's logs can't record anything relating to this event, and you have to look at some system logs to determine if this is the case. Details on how to do this are in How to Resolve JIRA Process being Terminated Unexpectedly in Linux.
Jira could be processing a large garbage collection of its memory. If this is a stop-the-world collection, the application becomes completely unresponsive until the Java garbage collection process can complete. I would recommend walking through Using Garbage Collection Logs to Analyze JIRA Application Performance. Personally, I like using gcviewer to review these logs files to see what kinds of stoppage time a Java application like Jira is recording, such as how often and for how long

Between those two possibilities that should cover the most common reasons Jira would appear unresponsive here. #1 is a condition in which you can only start Jira again to recover. Whereas #2 will eventually finish, it just can take some time depending on system performance. I'd also recommend that if you set the Xmx for Jira to be 4096m that you also set the Xms to be the same value. The reason for this is also covered in Why is it important to have xms and xmx value as same?. The short answer is, it tends to help overall performance to eliminate Java from having to do any kind of resizing on the heap itself (increases or decreases), as both event types cause a stop-the-world type event for an application like Jira.

If you have covered these steps above and are still seeing this same stuck thread problem, then I would want to recommend trying to Generating a thread dump. Or rather a series of thread dumps just before and during this the time when this happens to Jira. Sometimes we can learn more from the threads when the logs do not contain clear errors or warning.

Regards,

Andy

Shiroukoji

Thanks for your answer!

I don't have to stop Jira, it recovers after a little time.

I tried the recommendations you've linked, but it didnt't helped.

I did some thread-dumps but they don't give me any information I can work with.

Maybe you can help me. I uploaded them on my OneDrive, because I don't found an solution to attach a file to a post.

https://1drv.ms/u/s!AvTWLwQkyNe5hjgXEQ7kBAiMi8x7?e=ONMzAT

Andy Heinzer

Unfortunately, the thread dumps don't show any blocking, blocked, or deadlocked threads here that would be clear indicators of a problem bit of code. At least not in those pair of threads. Would it be possible for us to take a closer look at a support zip form this instance? Perhaps we can start looking there instead to see if there are other clues that might help us here.

Andy

vusxi1

27-May-2020 09:03:27.941 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [ajp-nio-8009

-exec-133] (id=[31725]) has been active for [120,702] milliseconds (since [5/27/20 9:01 AM]) to serve the same request for [https://jira.vcapps.org/] and may be stuck (configured threshold

for this StuckThreadDetectionValve is [120] seconds). There is/are [74] thread(s) in total that are monitored by this Valve and may be stuck.

java.lang.Throwable

Getting this logs and jira server went in hang mode,any one can help on this

Benjamin Yu

We have the same problem of stuck valve. We found when it reach 200 stuck threads, jira is not responsive, basically just crashes.

Here is the error:

may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

it normally take 3 hours from There is/are [1] thread(s) to There is/are [200] thread(s), then jira crashes. Our linux server has 64 gb memory, 8 CPUs.

We need help on this ASAP.

Thank you.

Luong La

I have same issue when ran Integrity checker
Error: "../secure/admin/IntegrityChecker.jspa] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds).

jira v9.5.0 #950000-sha1:bfd830f
with
OS: Ubuntu 20.04
Java Version 11.0.17
Java VM OpenJDK 64-Bit Server VM
Apache Tomcat: 9.0.68
Database type and version: postgres 14.4
Database driver PostgreSQL JDBC Driver 42.5.1

Benjamin Yu

Add more memory and CPU to my web server fixed this problem. You should try it.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

server

jira

jira-software

jira-software-server

docker