Thread dump analysis

Question

With some research around forums, I finally managed to get jstack to work to get a thread dump of a JAVA Process that is utilizing high CPU. Not only that, it has stopped writing to the designated log file and the process is haywire. I've checked that below thread is utilizing the highest CPU. Can someone help me to check on the root cause (as the logs are not showing anything):

Thread 30141: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150 (Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 (Compiled frame)
 - oracle.net.ns.Packet.receive() @bci=180, line=300 (Compiled frame)
 - oracle.net.ns.DataPacket.receive() @bci=1, line=106 (Compiled frame)
 - oracle.net.ns.NetInputStream.getNextPacket() @bci=48, line=315 (Compiled frame)
 - oracle.net.ns.NetInputStream.read(byte[], int, int) @bci=33, line=260 (Compiled frame)
 - oracle.net.ns.NetInputStream.read(byte[]) @bci=5, line=185 (Compiled frame)
 - oracle.net.ns.NetInputStream.read() @bci=70, line=102 (Compiled frame)
 - oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket() @bci=11, line=124 (Compiled frame)
 - java.util.LinkedList.linkLast(java.lang.Object) @bci=5, line=140 (Compiled frame)
 - java.util.LinkedList.add(java.lang.Object) @bci=2, line=336 (Compiled frame)
 - oracle.jdbc.driver.T4CTTIfun.receive() @bci=27, line=291 (Compiled frame)
 - oracle.jdbc.driver.T4CTTIfun.doRPC() @bci=56, line=192 (Compiled frame)
 - oracle.jdbc.driver.T4C8Oall.doOALL(boolean, boolean, boolean, boolean, boolean, oracle.jdbc.internal.OracleStatement$SqlKind, int, byte[], int, oracle.jdbc.driver.Accessor[], int, oracle.jdbc.driver.Accessor[], int, byte[], char[], short[], int, oracle.jdbc.driver.DBConversion, byte[], java.io.InputStream[][], byte[][][], oracle.jdbc.oracore.OracleTypeADT[][], oracle.jdbc.driver.OracleStatement, byte[], char[], short[], oracle.jdbc.driver.T4CTTIoac[], int[], int[], int[], oracle.jdbc.driver.NTFDCNRegistration) @bci=769, line=531 (Compiled frame)
 - oracle.jdbc.driver.T4CPreparedStatement.doOall8(boolean, boolean, boolean, boolean, boolean) @bci=749, line=207 (Compiled frame)
 - oracle.jdbc.driver.T4CPreparedStatement.executeForRows(boolean) @bci=226, line=1044 (Compiled frame)
 - oracle.jdbc.driver.OracleStatement.executeMaybeDescribe() @bci=179, line=1199 (Compiled frame)
 - oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout() @bci=150, line=1289 (Compiled frame)
 - oracle.jdbc.driver.OraclePreparedStatement.executeInternal() @bci=119, line=3584 (Compiled frame)
 - oracle.jdbc.driver.OraclePreparedStatement.executeQuery() @bci=13, line=3628 (Compiled frame)
 - oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery() @bci=4, line=1493 (Compiled frame)
 - org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery() @bci=12, line=96 (Compiled frame)
 - org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery() @bci=12, line=96 (Compiled frame)
 - org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery() @bci=12, line=96 (Compiled frame)
 - org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(java.sql.PreparedStatement) @bci=19, line=56 (Compiled frame)
 - org.hibernate.loader.Loader.getResultSet(java.sql.PreparedStatement, org.hibernate.engine.spi.RowSelection, org.hibernate.dialect.pagination.LimitHandler, boolean, org.hibernate.engine.spi.SessionImplementor) @bci=18, line=2040 (Compiled frame)
 - org.hibernate.loader.Loader.executeQueryStatement(java.lang.String, org.hibernate.engine.spi.QueryParameters, boolean, java.util.List, org.hibernate.engine.spi.SessionImplementor) @bci=85, line=1837 (Compiled frame)
 - org.hibernate.loader.Loader.executeQueryStatement(org.hibernate.engine.spi.QueryParameters, boolean, java.util.List, org.hibernate.engine.spi.SessionImplementor) @bci=10, line=1816 (Compiled frame)
 - org.hibernate.loader.Loader.doQuery(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters, boolean, org.hibernate.transform.ResultTransformer) @bci=44, line=900 (Compiled frame)
 - org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters, boolean, org.hibernate.transform.ResultTransformer) @bci=62, line=342 (Compiled frame)
 - org.hibernate.loader.Loader.doList(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters, org.hibernate.transform.ResultTransformer) @bci=34, line=2526 (Compiled frame)
 - org.hibernate.loader.Loader.doList(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters) @bci=4, line=2512 (Compiled frame)
 - org.hibernate.loader.Loader.listIgnoreQueryCache(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters) @bci=4, line=2342 (Compiled frame)
 - org.hibernate.loader.Loader.list(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters, java.util.Set, org.hibernate.type.Type[]) @bci=47, line=2337 (Compiled frame)
 - org.hibernate.loader.hql.QueryLoader.list(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters) @bci=19, line=495 (Compiled frame)
 - org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(org.hibernate.engine.spi.SessionImplementor, org.hibernate.engine.spi.QueryParameters) @bci=139, line=356 (Compiled frame)
 - org.hibernate.engine.query.spi.HQLQueryPlan.performList(org.hibernate.engine.spi.QueryParameters, org.hibernate.engine.spi.SessionImplementor) @bci=192, line=195 (Compiled frame)
 - org.hibernate.internal.SessionImpl.list(java.lang.String, org.hibernate.engine.spi.QueryParameters) @bci=49, line=1269 (Compiled frame)
 - org.hibernate.internal.QueryImpl.list() @bci=27, line=101 (Compiled frame)
 - org.hibernate.ejb.QueryImpl.getSingleResult() @bci=4, line=285 (Compiled frame)
 - com.dhl.odd.persistence.impls.UrlTokenServiceImpl.isTokenExist(java.lang.String) @bci=23, line=177 (Compiled frame)
 - com.dhl.odd.persistence.impls.UrlTokenServiceImpl.generateToken(java.lang.String) @bci=9, line=155 (Compiled frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=57 (Interpreted frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Compiled frame)
 - org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=7, line=317 (Interpreted frame)
 - org.springframework.aop.framework.JdkDynamicAopProxy.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=293, line=201 (Interpreted frame)
 - com.sun.proxy.$Proxy127.generateToken(java.lang.String) @bci=16 (Interpreted frame)
 - com.dhl.odd.batch.tasklets.GenerateUrlTasklet.getOddUrl(com.dhl.odd.apis.dtos.NotificationDto) @bci=164, line=93 (Interpreted frame)
 - com.dhl.odd.batch.tasklets.GenerateUrlTasklet.execute(org.springframework.batch.core.StepContribution, org.springframework.batch.core.scope.context.ChunkContext) @bci=107, line=57 (Interpreted frame)
 - com.dhl.odd.batch.tasklets.NotificationProcessorTasklet.execute(org.springframework.batch.core.StepContribution, org.springframework.batch.core.scope.context.ChunkContext) @bci=37, line=40 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=57 (Interpreted frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Compiled frame)
 - org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=7, line=317 (Interpreted frame)
 - org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint() @bci=12, line=190 (Interpreted frame)
 - org.springframework.aop.framework.ReflectiveMethodInvocation.proceed() @bci=19, line=157 (Interpreted frame)
 - org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(org.aopalliance.intercept.MethodInvocation) @bci=1, line=133 (Interpreted frame)
 - org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(org.aopalliance.intercept.MethodInvocation) @bci=75, line=121 (Interpreted frame)
 - org.springframework.aop.framework.ReflectiveMethodInvocation.proceed() @bci=101, line=179 (Interpreted frame)
 - org.springframework.aop.framework.JdkDynamicAopProxy.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=321, line=207 (Interpreted frame)
 - com.sun.proxy.$Proxy154.execute(org.springframework.batch.core.StepContribution, org.springframework.batch.core.scope.context.ChunkContext) @bci=20 (Interpreted frame)
 - org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(org.springframework.transaction.TransactionStatus) @bci=79, line=406 (Interpreted frame)
 - org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(org.springframework.transaction.TransactionStatus) @bci=2, line=330 (Interpreted frame)
 - org.springframework.transaction.support.TransactionTemplate.execute(org.springframework.transaction.support.TransactionCallback) @bci=38, line=133 (Interpreted frame)
 - org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(org.springframework.batch.repeat.RepeatContext, org.springframework.batch.core.scope.context.ChunkContext) @bci=58, line=271 (Interpreted frame)
 - org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(org.springframework.batch.repeat.RepeatContext) @bci=104, line=77 (Interpreted frame)
 - org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(org.springframework.batch.repeat.RepeatContext, org.springframework.batch.repeat.RepeatCallback, org.springframework.batch.repeat.support.RepeatInternalState) @bci=52, line=368 (Interpreted frame)
 - org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(org.springframework.batch.repeat.RepeatCallback) @bci=177, line=215 (Interpreted frame)
 - org.springframework.batch.repeat.support.RepeatTemplate.iterate(org.springframework.batch.repeat.RepeatCallback) @bci=10, line=144 (Interpreted frame)
 - org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(org.springframework.batch.core.StepExecution) @bci=75, line=257 (Interpreted frame)
 - org.springframework.batch.core.step.AbstractStep.execute(org.springframework.batch.core.StepExecution) @bci=87, line=198 (Interpreted frame)
 - org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call() @bci=8, line=139 (Interpreted frame)
 - org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call() @bci=1, line=136 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

Answer 1

You can't debug a high CPU usage from a single call stack. Since the thread has high CPU usage, it is likely that the call stack changes over time. Therefore you need multiple call stacks in order to tell what it is doing. Take several thread dumps and find the common part of the call stack.

The only exception to a changing call stack would be an endless loop inside a single function. That's unlikely here, since java.net.SocketInputStream is used in so many applications worldwide, someone would have noticed that already.

In this particular case, there's a database involved. Perhaps the SQL query just selects a large amount of data. Check if you can find out what it is transferring using Wireshark .

The next thing I noticed is org.springframework.batch.repeat.support.RepeatTemplate.iterate . I never used Spring, so I don't exactly know what it does, but it sounds much like it would repeat something several times. You may want to look into that as well.