How to access JMX interface in docker from outside?

Question

I am trying to remotely monitor a JVM running in docker. The configuration looks like this:

• machine 1: runs a JVM (in my case, running kafka) in docker on

Answer 1

To add some additional insights, I had some Docker port mappings in use, and none of the previous answers worked directly for me. After investigation, I found the answer here: How to connect with JMX from host to Docker container in Docker machine? to provide the required insights.

This is what I believe happens:

I set up JMX as suggested in other answers here:

-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.port=1098
-Dcom.sun.management.jmxremote.rmi.port=1098
-Djava.rmi.server.hostname=localhost
-Dcom.sun.management.jmxremote.local.only=false

Program flow:

• I run the Docker container and expose/map the port from host to container. Say I map port host:1099->container:1098 in Docker.
• I run the JVM inside the docker with the above JMX settings.
• The JMX agent inside the Docker container now listens to the given port 1098.
• I start JConsole on the host (outside Docker) with URL localhost:1099. I use 1099, since I used host:docker port mapping of 1099:1098.
• JConsole connects fine to the JMX agent inside Docker.
• JConsole asks JMX where to read the monitoring data.
• JMX agent responds with the configured information and address: localhost:1098
• JConsole now tries to connect to the given address localhost:1098
• This fails since port 1098 on localhost (outside Docker) is not listening. Port 1099 was mapped to Docker:1098. Instead of localhost:1098, JMX should tell JConsole to read monitoring information from localhost:1099, since 1099 was the port mapped from host to 1098 inside Docker container.

As a fix, I changed my host:docker port mapping from 1099:1098 to 1098:1098. Now, JMX still tells JConsole to connect to localhost:1098 for monitoring information. But now it works since the outside port is the same as advertised by JMX inside Docker.

I expect the same applies also for SSH tunnels and similar scenarios. You have to match what you configure JMX to advertise and what JConsole sees as the address space on the host where you run it.

Maybe it is possible to play a bit with the jmxremote.port, jmxremove.rmi.port, and hostname attributes to make this work using different port mappings. But I had the opportunity to use the same ports, so using them simplified it, and this works (for me).

Answer 2

For dev environment you can set java.rmi.server.hostname to the catch-all IP address 0.0.0.0

Example:

 -Djava.rmi.server.hostname=0.0.0.0 \
                -Dcom.sun.management.jmxremote \
                -Dcom.sun.management.jmxremote.port=${JMX_PORT} \
                -Dcom.sun.management.jmxremote.rmi.port=${JMX_PORT} \
                -Dcom.sun.management.jmxremote.local.only=false \
                -Dcom.sun.management.jmxremote.authenticate=false \
                -Dcom.sun.management.jmxremote.ssl=false

Answer 3

For completeness, the following solution worked. The JVM should be run with specific parameters established to enable remote docker JMX monitoring were as followed:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=<PORT>
-Dcom.sun.management.jmxremote.rmi.port=<PORT>
-Djava.rmi.server.hostname=<IP>

where:

<IP> is the IP address of the host that where you executed 'docker run'
<PORT> is the port that must be published from docker where the JVM's JMX port is configured (docker run --publish 7203:7203, for example where PORT is 7203). Both `port` and `rmi.port` can be the same. 

Once this is done you should be able to execute JMX monitoring (jmxtrans, node-jmx, jconsole, etc) from either a local or remote machine.

Thanks to @Chris-Heald for making this a really quick and simple fix!

Answer 4

I found it that trying to set up JMX over RMI is a pain, especially because of the -Djava.rmi.server.hostname=<IP> which you have to specify on startup. We're running our docker images in Kubernetes where everything is dynamic.

I ended up using JMXMP instead of RMI, as this only need one TCP port open, and no hostname.

My current project uses Spring, which can be configured by adding this:

<bean id="serverConnector"
    class="org.springframework.jmx.support.ConnectorServerFactoryBean"/>

(Outside Spring you need to set up your own JMXConncetorServer in order to make this work)

Along with this dependency (since JMXMP is an optional extension and not a part of the JDK):

<dependency>
    <groupId>org.glassfish.main.external</groupId>
    <artifactId>jmxremote_optional-repackaged</artifactId>
    <version>4.1.1</version>
</dependency>

And you need to add the same jar your your classpath when starting JVisualVM in order to connect over JMXMP:

jvisualvm -cp "$JAVA_HOME/lib/tools.jar:<your_path>/jmxremote_optional-repackaged-4.1.1.jar"

Then connect with the following connection string:

service:jmx:jmxmp://<url:port>

(Default port is 9875)

Answer 5

After digging around for quite a lot, I found this configuration

-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.port=1098
-Dcom.sun.management.jmxremote.rmi.port=1098
-Djava.rmi.server.hostname=localhost
-Dcom.sun.management.jmxremote.local.only=false

The difference with the other above is that java.rmi.server.hostname is set to localhost instead of 0.0.0.0