How to use JConsole to monitor Java apps remotely? (i.e. JMX Monitoring)

[Piyal Nanayakkara]

Hi

I have a java application running on Ubuntu server and I need to do JMX Monitoring using JConsole in my workstation.

What are the changes I need to do in Server level and client level?

I tried it with default settings and I am getting "Connection Failed: non-JRMP server at remote endpoint" error. Please help

Regards

Piyal

[Rafael Pinto Sperafico]

Hi @Piyal Nanayakkara ,

You will need to add/append com.sun.management.jmxremote.* to the application you want to connect to. Would you mind sharing which application you will be monitoring so we can provide you with more details on how to set it up?

You could append the following properties to the application you are going to monitor, e.g

JAVA_OPTS=-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=8123 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false

Then connect JConsole to hostname:port

Kind regards,
Rafael

[Piyal Nanayakkara]

Thanks Rafael.

I am trying to add JMX Monitoring to the JIRA Software version 7.6.11 which runs on an Ubuntu Server and my JConsole runs on Windows10 Enterprise.

So how can I add the given properties to this application? Please provide step by step details.

Regards

Piyal

[Rafael Pinto Sperafico]

Hi @Piyal Nanayakkara ,

You can find the steps described in https://confluence.atlassian.com/adminjiraserver/live-monitoring-using-the-jmx-interface-939707304.html

Kind regards,
Rafael

[Piyal Nanayakkara]

Hi Rafael

I tried with the steps given. But JConsole remote non secure connection to Jira server fails.

Here is my initial Setenv.sh related section:

CATALINA_OPTS="${GC_JVM_PARAMETERS} ${CATALINA_OPTS}"
export CATALINA_OPTS

Then I modified it as below and restarted Jira Server.

CATALINA_OPTS="${GC_JVM_PARAMETERS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=false ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.ssl=false ${CATALINA_OPTS}"
export CATALINA_OPTS

In the JConsole (I have it in C:\Program Files\Java\jdk1.8.0_162 folder of my workstation), I can see below input fields for Remote access:

- Remote proccess    (I provided the JIRA Server IP and port as 8099)

- User name and Password  (since jmxremote.authenticate=false, I assumed we dont need to provide the values.- please confirm. If we need to supply it, what are the values we need to give?)

Could you please provide your directions to resolve this?

Regards

Piyal

[Piyal Nanayakkara]

Additionally I verified that 8099 port is open using telnet.  Note that <Server IP> represents the JIRA Server's IP.

[jira@vro-safe-jira-uat bin]$ telnet <Server IP> 8099
Trying <Server IP>...
Connected to <Server IP>
Escape character is '^]'.

[Piyal Nanayakkara]

When running JConsole without credentials, I can see below JConsole Log initially:

Exception in thread "AWT-EventQueue-0" java.awt.IllegalComponentStateException: component must be showing on the screen to determine its location
at java.awt.Component.getLocationOnScreen_NoTreeLock(Component.java:2062)
at java.awt.Component.getLocationOnScreen(Component.java:2036)
at javax.swing.text.JTextComponent$InputMethodRequestsHandler.getTextLocation(JTextComponent.java:4643)
at sun.awt.im.InputMethodContext.getTextLocation(InputMethodContext.java:278)
at sun.awt.windows.WInputMethod$1.run(WInputMethod.java:588)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:90)
at java.awt.EventQueue$4.run(EventQueue.java:733)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
java.lang.SecurityException: Expecting a sun.rmi.server.UnicastRef2 remote reference in stub!
at javax.management.remote.rmi.RMIConnector.checkStub(RMIConnector.java:1891)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:295)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:355)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:294)

Then, when I click on "Insecure connection" button, JConsole connection failed with below JConsole log:

java.rmi.ConnectException: Connection refused to host: 10.52.211.191; nested exception is:
java.net.ConnectException: Connection timed out: connect
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy0.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:355)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:294)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 11 more

Please advise.

Regards

Piyal

[Rafael Pinto Sperafico]

Hi @Piyal Nanayakkara ,

Perhaps you could try appending the following property:

java.rmi.server.hostname – The IP address of the host machine where your Java application is running. This default value is the IP address of the local host. If you only have a single network interface, then you do not need to set this property. But in case, you have multiple active network adapters, then you have to specify this value as the Java program can pick IP address of any one of the network adapter.

CATALINA_OPTS="${GC_JVM_PARAMETERS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=false ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.ssl=false ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.rmi.server.hostname=192.168.1.2 ${CATALINA_OPTS}"
export CATALINA_OPTS

Please, note that 192.168.1.2 in above is just an IP address example to the network adapter where the application is running (e.g Jira).

Another thing to consider is Enabling JMX monitoring in Jira as described in https://confluence.atlassian.com/adminjiraserver/live-monitoring-using-the-jmx-interface-939707304.html

Alternative, as a troubleshooting process, you could attempt on connecting JMX from the same box where Jira is running, so you can confirm it is working.

Hope the above helps.

Kind regards,
Rafael

[Piyal Nanayakkara]

Hi Rafael

Thanks for the prompt reply.

I applied the proposed changes but still it fails. Here are the 2 mian errors in JConsole log:

1) java.lang.SecurityException: Expecting a sun.rmi.server.UnicastRef2 remote reference in stub!

2) java.rmi.ConnectException: Connection refused to host: 10.52.211.191; nested exception is: java.net.ConnectException: Connection timed out: connect.

I have already enabled  JMX monitoring in Jira.

Also I tried to  connecting JMX from the same box where Jira is running, but its a terminal server there is no GUI in the Jira Server. (everything is done using command line interface)

Seems like real issue is "Connection timed out" - So, do you have any suggestions to fix it? :-)

Regards

Piyal

[Rafael Pinto Sperafico]

Hi @Piyal Nanayakkara ,

Just to confirm, can you access Jira from web browser using the IP address 10.52.211.191 (e.g http://10.52.211.191:8080) from the Windows machine you are trying to connect JConsole?

Kind regards,
Rafael

[Piyal Nanayakkara]

Yes, I can access Jira from web browser using the IP address 10.52.211.191

Regards

Piyal

[Daniel Corrigan]

@Piyal Nanayakkara 

This works for me in setenv.sh

Adjust for your server IP. 

CATALINA_OPTS="-Dcom.sun.management.jmxremote ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.ssl=false ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=false ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.rmi.server.hostname=10.40.8.206 ${CATALINA_OPTS}"