Blog: Profiling remote JVM using VisualVM

Profiling is essential when you want to improve performance of an application. To properly profile, you should minimize the differences between an actual production situation and the situation you are profiling. This means that your test data should be the same but also the hardware should be as similar as possible. The best possible scenario is to actually profile the production environment. You can run most profiling tools (e.g. VisualVM) on your local development machine and have it create a connection to the production environment.

To do this, you’ll need to expose the JVM somehow so you can reach it from your local machine. There are two ways of doing it, by starting your application with some special JMX properties, or by running jstatd on that machine.

If you can modify the start up scripts of your application and actually restart your application, the JMX option is possible. However, in my experience, it is often quite difficult to modify start-up scripts as they are almost always managed by a different team. And, often restarting the application for the new parameters to take effect is not feasible.

Another route is running jstatd. This standard java application runs an rmi registry and allows VisualVM to connect to it. In theory it is just two simple steps:

1. On the remote machine, start jstatd 

2. On your dev machine, start VisualVM and add a new remote host 

On some operating systems (e.g. Ubuntu Linux) jstatd won’t start up but instead throws an exception like:

 

Could not create remote object
access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")

This can be fixed (as described by the man page of jstatd) by creating a security policy file, giving jstatdall access to everything. Needless to say, if this is your production environment, you might want to be a bit careful and make sure it is not accessible for the whole world. Create a jstatd.all.policy file with the following content:

grant codebase "file:/opt/java/jdk1.7.0_21/lib/tools.jar" { permission java.security.AllPermission; };

Note that in my experience relative paths to tools.jar will not work, hence the hardcoded path to the running jvm. You should then start jstatd using this policy file:

jstatd -J-Djava.security.policy=/home/user/jstatd.all.policy

If VisualVM doesn’t list the remote jvm’s, here are some tips in getting it working: 

– Firewall issues: can you telnet from your dev machine to the jstatd port (default 1099)? If not, you’ll probably need to create ssh tunnels. Don’t forget that jstatd opens a second (randomly assigned) port as well which you’ll need to forward. 
– Binding issues: sometimes jstatd doesn’t bind to the correct ip address. You can force binding it to a specific ip using something like

-J-Djava.rmi.server.hostname=10.1.1.123. 

– IPv6 problems: On rare occassions it can help to force jstatd and/or VisualVM to use IPv4:

-J-Djava.net.preferIPv4Stack=true 

– Enable logging: If the steps above don’t work, you could try to enable some extra logging to jstatd to see if you can trace the problem:

-J-Djava.rmi.server.logCalls=true 

 
Some references: