We get this message: Memory Utilization is 70.572%, crossed warning (65) or critical (70) threshold.

[Cesar Rouillon]

 

atlassian-jira.log-5.zip ‏509 KB

atlassian-jira.log-2.zip ‏605 KB

logs jira.zip ‏5 KB

atlassian-jira.log-4.zip ‏513 KB

atlassian-jira.log-3.zip ‏685 KB

Consumo_Recursos_Servidor_Jira.jpg ‏46 KB

atlassian-jira.log-1.zip ‏669 KB

[Cesar Rouillon]

What can we do?, is possible that we can configured some parameter in the JIRA or database?

Thanks

[Nic Brough [Adaptavist]]

Where are you seeing this, and why are suspecting a parameter rather than it being caused by load on your system?

[Cesar Rouillon]

Our monitor tool's "OracleCloud" on application server sent us this message:

Host=sp12app09.edyficar.local

Target type=Host
Target name=sp12app09.edyficar.local
Incident creation time=Feb 18, 2016 2:23:22 PM PET
Last updated time=Feb 18, 2016 2:23:24 PM PET
Message=Memory Utilization is 70.572%, crossed warning (65) or critical (70) threshold.
Severity=Critical
Incident ID=1421
Event count=1
Incident Status=New
Escalated=No
Priority=High
Incident owner=ADM_ADMINISTRADOR
Incident Acknowledged By Owner=No
Categories=Capacity
Rule Name=Rule_OS,RULE_OS
Rule Owner=SYSMAN
Notification Count=1

We don't know what is the reason for this, is possible that a hide proccess is running?

Thanks!.

[Nic Brough [Adaptavist]]

Not really.  The service is using more memory than the threshold you've set in monitoring.  That doesn't tell you what it's doing or why its doing it. 

The most likely explanation is that the system has come under additional load.  You need to investigate the pattern of usage of your JIRA system.

[Cesar Rouillon]

In this image you can look the %CPU of JIRA system.

[Nic Brough [Adaptavist]]

I'm sorry, I don't mean the resource consumption, I mean you need to look at what your users and automated processes are doing that might be putting load on the system.

[Cesar Rouillon]

We don't know why the consume of JIRA in the application server is to much (see the image "Consumo_Recursos_Servidor_Jira.jpg"), beacuse in the database the consume is low. 

Can you get us a recommendation.

[Nic Brough [Adaptavist]]

Yes, you need to look at what your users and/or automated interfaces are doing to JIRA to make it need to use that resource.

The numbers you are quoting indicate load.  They tell us nothing about how your usage is causing that load.

[Nic Brough [Adaptavist]]

One thing that may help is a "thread trace" - you wait until the threshold is crossed again, and then go on to the server and run "jstack PID > outfile", where PID is the process ID of the Tomcat running JIRA and outfile is the log name you want.  When that finishes, the outfile will contain a long list of everything JIRA was doing when you ran it.

[Cesar Rouillon]

Dears,

Please your help, until now we get memory utilization critical in the server of jira, we used the command "thread trace", the result is the next:

Deadlock Detection:

No deadlocks found.

Thread 3109: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run() @bci=10, line=565 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3108: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- org.apache.jk.common.ChannelSocket.accept(org.apache.jk.core.MsgContext) @bci=46, line=311 (Interpreted frame)
- org.apache.jk.common.ChannelSocket.acceptConnections() @bci=72, line=668 (Interpreted frame)
- org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(java.lang.Object[]) @bci=4, line=879 (Interpreted frame)
- org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=167, line=690 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3107: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=26, line=662 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3106: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=26, line=662 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3105: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=26, line=662 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3101: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- sun.security.ssl.SSLServerSocketImpl.accept() @bci=42, line=317 (Interpreted frame)
- org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(java.net.ServerSocket) @bci=3, line=178 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run() @bci=47, line=352 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3100: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(java.net.ServerSocket) @bci=1, line=61 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run() @bci=47, line=352 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3099: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run() @bci=22, line=1631 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 3017: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=28, line=526 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)

Thread 2074: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
- org.logicalcobwebs.proxool.PrototyperThread.doWait() @bci=1, line=69 (Compiled frame)
- org.logicalcobwebs.proxool.PrototyperThread.run() @bci=140, line=58 (Compiled frame)

Thread 2073: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.logicalcobwebs.proxool.HouseKeeperThread.run() @bci=71, line=46 (Interpreted frame)

Thread 2072: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- com.icesoft.util.MonitorRunner$1.run() @bci=14, line=21 (Interpreted frame)

Thread 2065: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Listener$1.run() @bci=62, line=221 (Interpreted frame)

Thread 2036: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- org.apache.axis.monitor.SOAPMonitorService$ServerSocketThread.run() @bci=9, line=189 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 1807: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() @bci=1, line=52 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() @bci=55, line=388 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=360 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 1800: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() @bci=55, line=388 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=360 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 1797: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() @bci=55, line=388 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=360 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 1727: (state = BLOCKED)

Thread 1711: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=189 (Compiled frame)

Thread 1710: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)

Thread 1688: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- org.apache.catalina.core.StandardServer.await() @bci=164, line=431 (Interpreted frame)
- org.apache.catalina.startup.Catalina.await() @bci=4, line=676 (Interpreted frame)
- org.apache.catalina.startup.Catalina.start() @bci=194, line=628 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=57 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Interpreted frame)
- org.apache.catalina.startup.Bootstrap.start() @bci=37, line=289 (Interpreted frame)
- org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=131, line=414 (Interpreted frame)

Please your help..

Greetings.

César

[Nic Brough [Adaptavist]]

Nothing particularly untoward in there, although it does look far too short to be the full thread trace.  It doesn't really point to anything, but it was worth a try.  A thread dump analyser might help read the full one though.

I'm afraid you need to do the analysis on the front end - look at what your users are doing, read the application log to see what is in there at times when the memory usage is high, and monitor the server at those times too (is the disk thrashing?  The database?  The network? etc)

[Cesar Rouillon]

Where can I find the application log?, what is the directory?

[Nic Brough [Adaptavist]]

It's in <JIRA home>/logs - the main one is atlassian-jira.log, but there may be others in there that are useful.  It might also be worth looking at the application server (Tomcat) log - that's in <JIRA install>log and the main one is catalina.out

[Cesar Rouillon]

I add the logs (), please your help.

The last erros was in:

Lunes, 15 de febrero de 2016 04:50 p.m.

Miércoles 17 de febrero de 2016 05:45 p.m  

Martes, 23 de febrero de 2016 04:20 p.m.

Miércoles, 02 de marzo de 2016 04:00 p.m.

[Nic Brough [Adaptavist]]

That's the application server (Tomcat) log.  No faults in there, so it is healthy.

What does the application log say?

[Cesar Rouillon]

I add the appplication logs:

, , , , .

Please your help.

Thanks.

[Nic Brough [Adaptavist]]

I don't have time to work through all of that for you.  What does it say is happening around the points of high memory load?