Enabling JMX with SSL?

Christopher Kucera March 23, 2021

I was setting up some JMX monitoring on Jira Server, and I'd like to enable SSL for JMX itself.  A non-SSL setup works just fine, with this in my `bin/setenv.sh`:

CATALINA_OPTS="-Dcom.sun.management.jmxremote ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/usr/local/jira/jmxremote.password ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.ssl=false ${CATALINA_OPTS}"
export CATALINA_OPTS

However, nothing I've done seems to get SSL working properly.  One initial wrinkle, just to help out anyone else who might run into this, is that I did need to add "&useSSL=false" to our MySQL 8.0 database connection string inside dbconfig.xml, after setting the SSL parameters, otherwise the database connection would fail and Jira wouldn't start up at all.  Once I'd gotten that added, though, the following would at least have port 8099 start up listening on SSL, using the correct certificate:

CATALINA_OPTS="-Dcom.sun.management.jmxremote ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/usr/local/jira/jmxremote.password ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.ssl=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dcom.sun.management.jmxremote.registry.ssl=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Djavax.net.ssl.keyStore=/usr/local/jira/servercert.p12 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djavax.net.ssl.keyStoreType=PKCS12 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djavax.net.ssl.keyStorePassword=<keystorepass> ${CATALINA_OPTS}"

(substituting the real keystore pass on that last one, of course)  The "-Dcom.sun.management.jmxremote.registry.ssl=true" line was definitely required to make it work.  At this point, I could run something like "openssl s_client -showcerts -connect hostname:8099" and it would show that port 8099 was using the SSL cert properly.

However, no JMX client I tried could actually connect to it.  Here's JConsole's error:

java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
java.net.SocketException: Connection reset
at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:300)
at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at jdk.jconsole/sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:234)
at jdk.jconsole/sun.tools.jconsole.ProxyClient.<init>(ProxyClient.java:127)
at jdk.jconsole/sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:475)
at jdk.jconsole/sun.tools.jconsole.JConsole$3.run(JConsole.java:524)
Caused by: java.net.SocketException: Connection reset
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:982)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263)
at java.base/java.io.DataInputStream.readByte(DataInputStream.java:269)
at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:239)
... 7 more
java.rmi.ConnectIOException: non-JRMP server at remote endpoint
at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:244)
at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at jdk.jconsole/sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:234)
at jdk.jconsole/sun.tools.jconsole.ProxyClient.<init>(ProxyClient.java:127)
at jdk.jconsole/sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:475)
at jdk.jconsole/sun.tools.jconsole.JConsole$3.run(JConsole.java:524)

Some commandline tools also ended up throwing the "java.rmi.ConnectIOException: non-JRMP server at remote endpoint" error as well.  So even though it's listening on SSL, it doesn't seem to be a valid endpoint anymore.

I've tried a few other arguments as well, none of which have seemed to make any difference, such as:

CATALINA_OPTS="-Dcom.sun.management.jmxremote.ssl.need.client.auth=false ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.security.policy=jmx.policy ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.rmi.server.hostname=<hostname> ${CATALINA_OPTS}"

(substituting in the real server hostname in the last one there).  I'd also tried adding in the various "javax.net.ssl.trustStore" parameters, even though the default installed JRE trust stores looked totally fine, and that didn't change anything.  (I didn't really expect it to, since the SSL portion seemed to be working fine anyway.)

Any ideas?  Has anyone actually gotten JMX to work over SSL in Jira?  Practically everything I've found online is folks running it without SSL.

Thx!

1 answer

0 votes
Pascal Robert October 29, 2022

I have the same issue.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
VERSION
8.13.3
TAGS
AUG Leaders

Atlassian Community Events