Profiling a running Java application in command line

Question

I profile running Java applications often with VisualVM but it needs X to run on the machine.

I know I can connect through management port but that will be an offline sampled profiling which is not enough for me.

So I'm looking for a solution with which I can profile the CPU usage of the methods of a running Java application from command-line. It's enough for me to collect data on the server and then the collected data can be analyzed on a different machine.

Update:

It seems I need to be more specific. I want to profile a running Java application from command line, I don't want to stop it and rerun it.

Answer 1

The jvmtop application is a convenient tool for profiling from the commandline. No need to stop the jvm. Usage:

jvmtop.sh --profile <PID>

Will give you output like this which will be updating while the app runs:

  Profiling PID 24015: org.apache.catalina.startup.Bootstrap
  36.16% (    57.57s) hudson.model.AbstractBuild.calcChangeSet()
  30.36% (    48.33s) hudson.scm.SubversionChangeLogParser.parse()
   7.14% (    11.37s) org.kohsuke.stapler.jelly.JellyClassTearOff.parseScript()
  ...

The advantage is that it does not take the use of instrumentation. The classes of the to-be-profiled jvm will not be altered.

If you are looking for something more visual then have a look at jvm-mon which is based on jvmtop

Answer 2

Looks like the "built-in" way to profile a java app from the command line is to start it with profiling command line parameters, like this

$ java -Xrunhprof:cpu=samples,file=myprogram.hprof ...

Then examine the file "myprogram.hprof" with some GUI tool (or web server tool like jhat ) or command line tool after the process exits (and the file is created at that time).

If you use the "QUIT" signal trick, mentioned https://stackoverflow.com/a/2344436/32453 then you can generate a file at will without exiting the JVM (it appears to append to the previous output file). Or wait until the process exits and it will generate the file.

This (built-in) profiler does a sample infrequently so typically low slowdown/impact overall.

ref: http://web.archive.org/web/20160623224137/https://thunderguy.com/semicolon/2004/04/18/profiling-a-java-program-easily/

You could also just do the "poor man's profiler" by collecting lots of jstacks and dumping them into ex: a flamegraph or some other analyzer/conglomerator...

Answer 3

Can you collect 10 or 20 stack samples with jstack ? Then if Foo is a method, its overall time usage is the fraction of samples containing it. Its CPU usage is the fraction of those samples that don't terminate in I/O or a system call. Its "self time" is the fraction of samples in which it itself is the terminus.

I don't need anything pretty. I either run it under the IDE and collect them that way, or use something like jstack that snapshots the stack of a running app.

That's the random-pause technique.

Answer 4

We have used hprof on our servers and it definitely is better than sysouts in case you can't run a full fledged VisualVM session.

Examples of using hprof are plenty out there:

• http://www.javalobby.org/java/forums/t19612.html
• https://coderwall.com/p/il3h4g/easy-profiling-java-from-command-line
• http://www.brendangregg.com/blog/2014-06-09/java-cpu-sampling-using-hprof.html

Answer 5

The most precise profiling can be achieved with https://github.com/jvm-profiling-tools/async-profiler .

This project is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on HotSpot JVM.

Here is my script to install and run it from command-line:

async-profiler.sh

if [ ! -d profiler ]; then
  mkdir profiler && cd profiler && curl -L https://github.com/jvm-profiling-tools/async-profiler/releases/download/v1.6-ea/async-profiler-1.6-ea-linux-x64.tar.gz | tar xvz
  echo 1 > /proc/sys/kernel/perf_event_paranoid
  echo 0 > /proc/sys/kernel/kptr_restrict
  #apt install openjdk-8-dbg
else
  cd profiler
fi

#jps

./profiler.sh -d 60 -f dump_`date +%Y-%m-%d_%H-%M-%S`.jfr `jps -q`

It assumes that app is run under same user and there is a single java process PID to be listed by jps. Profiling duration is 60 seconds.

No modification of app's startup options or app restart is needed.

GUI for examining dumps is built-in into IntelliJ IDEA Ultimate: https://www.jetbrains.com/help/idea/cpu-profiler.html .

Answer 6

One way to profile an "already started" JVM is to aggregate multiple jstacks taken over time.

You can for instance parse and display them as a FlameGraph (see details at the various answers for that link, I won't redundantly include them here).

Answer 7

You can run most commercial profilers remotely so an agent is run on the server then connect to that agent through a client on your dev machine. My absolute favorite profiler is JProfiler. It's fairly reasonable purchase, and it's very stable (which not all commercial profilers that's true).

http://www.ej-technologies.com/products/jprofiler/overview.html

Other commercial profilers that are stable, but not my favorite are YourKIT.

http://www.yourkit.com/

Those smaller vendors make good tools. These tools will provide you tons of information about method timings, memory use, GC, etc. Much more than jconsole.