Cpu utilization goes really high with for asynch api

Question

I have an api that is used to store data in DynamoDB , I am using google's ListeningExecutorService to submit jobs, which eventually sends the data to DB. but the cpu utilization is going really high and i am unable to debug why.

Here is the code:

@Timed(AuditConstants.CRITICAL_TIMER_PREFIX + "sendMessage")
public void save(Audit audit) {
    Callable callable = new CallableResult(auditRepository,auditDataUtils,audit);

    ListenableFuture<Result> future = executorService.submit(callable);

    Futures.addCallback(future, new FutureCallback<Result>() {
        @Override
        public void onSuccess(@Nullable Result result) {
            log.info("Successfully sent audit with uuid {} to Dynamo DB", audit.getAuditId());
        }

        @Override
        public void onFailure(Throwable throwable) {
            log.info(ERROR_WHILE_SENDING_THE_MESSAGE, throwable);
            handleException(audit, throwable);
        }
    });

}

private void handleException(Audit audit, Throwable ex) {

    log.info("Failed talking to Dynamo DB, writing audit with uuid {} to file", audit.getAuditId(), ex);
    producerAuditEventsLogger.logToFile(audit);
}

CallableResult job

@Override
    public Result call() throws Exception {
        try {
            AuditData auditData = auditDataUtils.getAuditData(audit);
            auditRepository.saveAuditData(auditData);
            return new Result(SUCCESS);
        } catch (Exception e) {
            throw new Exception("There was an error while writing to DB",e);
        }
    }

The CPU Utilization is reaching upto 70% forr 3 Nodes(c4x large ) for 1300 tps.

The response time is very low ~20ms

Kindly guide me how can I debug, and what are the recommended ways to lower cpu.

Thanks

Answer 1

You can try in following directions:

1. Check if you are doing too much work inside the thread

2. See if DB operations are taking too long to execute or are the ones taking up CPU (if on same node).

3. Take a look at visual VM snapshot and try to figure out which thread is taking more CPU.

4. Or your job is scheduled to run at a very high frequency causing too many thread to be created.

To me it looks like your DB operations are causing it.