在生产中进行定期线程转储是否昂贵？

Question

We have a Java app in production where a certain thread is stalling/backed up. The thread is reading off a queue and we measure how long it takes for an inserted task to be processed . What's the best way to go about debugging the root cause. Would taking peroidic thread dumps (via script) every minute or so provide more information? What have others done to debug such situations.

Answer 1

In the system I work on, we monitor the time it takes for tasks to execute. If this time exceeds X amount of time, we trigger a thread dump (programatically, from the point where we measure the time, so not an external script), followed by another thread dump a few seconds later. This threshold X should be a relatively large number, in our case it is 5 minutes. If this occurs, we can assume that the system is not "just slow", but something bad happened, like a deadlock or an extremely long blocking call.

So, to answer part of your question: yes, periodic thread dumps could help, but only if the dumps are from the exact moment the event you are looking for occured. If you just generate a thread dump every 10 seconds, finding the correct dump could be a pain... Unless if you are looking for a deadlock, tools can help with that. I can't answer the performance part of your question.

Answer 2

Assuming your thread is pulling tasks off a simple concurrent Queue implementation I'd start by checking whether garbage collection is the culprit. If you're not already doing so you'll want to add command line options to turn on GC logging:

-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCDateStamps -Xloggc:<some-file>

If you're sure it's not garbage collection then you could consider using something like jHiccup to monitor stalls particularly if your application is running in a virtualized environment.

Answer 3

Taking periodic thread dumps in production is an expensive operation for sure. Since you are giving additional task to JVM to print all the current execution stacks of all the threads which are spawned by the JVM.

If you have the access to the code, i would advice you to either have additional logging to print your performance logs and timing data.

If you don't have access to code, I would recommend using an APM tool like dynatrace, appdynamics or anything which is available to debug the time taking method or third party call.

Hope this helps!

Regards, Eby J