如何确定Java进程的CPU 100％利用率

Question

I have a Java program utilizing nearly 100% of the CPU. I have tried taking the java thread dump but I am unable to locate any issue with the code. Can anyone help me locate the issue

    2015-11-03 16:24:45
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):

"RMI TCP Connection(5)-104.130.180.175" #3298 daemon prio=5 os_prio=0 tid=0x000000001633a800 nid=0xf64 in Object.wait() [0x000000001923c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:449)
    - locked <0x00000000c0a84b30> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
    at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:227)
    at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:274)
    at javax.management.remote.rmi.RMIConnectionImpl$3.run(RMIConnectionImpl.java:1268)
    at javax.management.remote.rmi.RMIConnectionImpl$3.run(RMIConnectionImpl.java:1266)
    at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1272)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
    at sun.rmi.transport.Transport$1.run(Transport.java:200)
    at sun.rmi.transport.Transport$1.run(Transport.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$254(TCPTransport.java:683)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1991638838.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x00000000c0a84ed0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"RMI TCP Connection(3)-104.130.180.175" #3296 daemon prio=5 os_prio=0 tid=0x000000001633b800 nid=0xf9c runnable [0x000000001903e000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    - locked <0x00000000c0a869d8> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$254(TCPTransport.java:683)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1991638838.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x00000000c0a86c28> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"RMI TCP Connection(2)-104.130.180.175" #3295 daemon prio=5 os_prio=0 tid=0x000000001633d000 nid=0xd38 runnable [0x0000000018f3e000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    - locked <0x00000000c0a87130> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$254(TCPTransport.java:683)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1991638838.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x00000000c0a87270> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"JMX server connection timeout 3294" #3294 daemon prio=5 os_prio=0 tid=0x000000001633d800 nid=0xf14 in Object.wait() [0x0000000016f6f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
    - locked <0x00000000c0a8fe08> (a [I)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"RMI Scheduler(0)" #3293 daemon prio=5 os_prio=0 tid=0x0000000016339000 nid=0x7dc waiting on condition [0x0000000016d6f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c0a90128> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"RMI TCP Accept-0" #3289 daemon prio=5 os_prio=0 tid=0x00000000163c1000 nid=0xd5c runnable [0x000000001696e000]
   java.lang.Thread.State: RUNNABLE
    at java.net.DualStackPlainSocketImpl.accept0(Native Method)
    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
    - locked <0x00000000c0a9aa68> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"MySQL Statement Cancellation Timer" #36 daemon prio=5 os_prio=0 tid=0x00000000163bd000 nid=0xc08 in Object.wait() [0x0000000015a8f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000c069e780> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:502)
    at java.util.TimerThread.mainLoop(Timer.java:526)
    - locked <0x00000000c069e780> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
    - None

"Thread-14" #35 prio=5 os_prio=0 tid=0x00000000163bf800 nid=0x7b8 runnable [0x000000001746f000]
   java.lang.Thread.State: RUNNABLE
    at java.net.DualStackPlainSocketImpl.accept0(Native Method)
    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
    - locked <0x00000000c069eae8> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at com.teleai.smsg.gateway.SocketServer.run(SocketServer.java:20)

   Locked ownable synchronizers:
    - None

"Monitor Thread" #34 prio=5 os_prio=0 tid=0x00000000163bf000 nid=0x43c waiting on condition [0x000000001736f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.teleai.smsg.gateway.Monitor.run(Monitor.java:24)

   Locked ownable synchronizers:
    - None

"Timer-0" #33 daemon prio=5 os_prio=0 tid=0x00000000163bb000 nid=0x6e0 in Object.wait() [0x000000001716f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    - locked <0x00000000c069f678> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
    - None

"QueueProcessor Thread Telco: 236" #32 prio=5 os_prio=0 tid=0x00000000163bb800 nid=0xdb8 runnable [0x000000001706e000]
   java.lang.Thread.State: RUNNABLE
    at com.teleai.smsg.gateway.QueueProcessor.run(QueueProcessor.java:92)
    - locked <0x00000000c069fa50> (a com.teleai.smsg.gateway.QueueProcessor)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"QueueProcessor Thread Telco: 162" #28 prio=5 os_prio=0 tid=0x00000000163ba000 nid=0xe98 runnable [0x0000000016e6f000]
   java.lang.Thread.State: RUNNABLE
    at com.teleai.smsg.gateway.QueueProcessor.run(QueueProcessor.java:92)
    - locked <0x00000000c06a6440> (a com.teleai.smsg.gateway.QueueProcessor)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"QueueProcessor Thread Telco: 158" #24 prio=5 os_prio=0 tid=0x00000000163e1000 nid=0x93c runnable [0x0000000016c6f000]
   java.lang.Thread.State: RUNNABLE
    at com.teleai.smsg.gateway.QueueProcessor.run(QueueProcessor.java:92)
    - locked <0x00000000c06a7050> (a com.teleai.smsg.gateway.QueueProcessor)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"QueueProcessor Thread Telco: 4" #20 prio=5 os_prio=0 tid=0x0000000016519800 nid=0xc7c runnable [0x0000000016a6e000]
   java.lang.Thread.State: RUNNABLE
    at com.teleai.smsg.gateway.QueueProcessor.run(QueueProcessor.java:92)
    - locked <0x00000000c06c1f80> (a com.teleai.smsg.gateway.QueueProcessor)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"DestroyJavaVM" #14 prio=5 os_prio=0 tid=0x0000000001180800 nid=0xe6c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Wrapper-Connection" #13 daemon prio=10 os_prio=2 tid=0x00000000154c3800 nid=0xa48 runnable [0x0000000015b8f000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.net.SocketInputStream.read(SocketInputStream.java:223)
    at java.io.DataInputStream.readByte(DataInputStream.java:265)
    at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3737)
    at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

"Wrapper-Control-Event-Monitor" #11 daemon prio=5 os_prio=0 tid=0x000000001541b800 nid=0xfbc waiting on condition [0x000000001598e000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.tanukisoftware.wrapper.WrapperManager$3.run(WrapperManager.java:731)

   Locked ownable synchronizers:
    - None

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00000000151c0800 nid=0x6b8 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000000013952000 nid=0xb10 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x000000001394e000 nid=0xd0 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000001395b000 nid=0x6b4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000013959000 nid=0x74 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000013957800 nid=0x478 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000000001277000 nid=0x7a8 in Object.wait() [0x0000000014c9e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    - locked <0x00000000c07b0768> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

   Locked ownable synchronizers:
    - None

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x000000000126e800 nid=0x8e4 in Object.wait() [0x0000000014b9f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
    - locked <0x00000000c078db68> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
    - None

"VM Thread" os_prio=2 tid=0x0000000013917800 nid=0x958 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000001196800 nid=0xc3c runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000001198000 nid=0xa80 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000001199800 nid=0x474 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x000000000119b800 nid=0x8c8 runnable 

"VM Periodic Task Thread" os_prio=2 tid=0x00000000139ae800 nid=0xb7c waiting on condition 

JNI global references: 255

I have also use process explorer to identify the threads, below is a snapshot

I have used Visual VM and got following results

Following is the situation around threads

I am still clueless about the direction to follow, can anyone point me in the right direction.

I have tried on more thing is identifying the thread ID inside the thread dump by converting it to hexa after getting the thread id from process explorer. This way I am able to identify the exact line number inside the code.

Answer 1

The CPU usage screenshot shows 4 threads using CPU and the thread dump shows 4 threads at com.teleai.smsg.gateway.QueueProcessor.run(QueueProcessor.java:92) . More than a coincidence, I'm sure it is the culprit.

Answer 2

I have used process explore to identify the thread ID as mentioned in process explorer above and converted them to hexa decimal as below. 3196 ‭C7C‬ 3736 ‭E98‬ 3512 ‭DB8‬ 2364 ‭93C‬

This helps me locate the exact line number within the dump ie QueueProcessor.java:92)