Java - Async - Thread pool

Question

I'm trying to understand the benefits of async in java.

Scenario 1: I have a spring boot web app deployed to tomcat, with tomcat min and max threads both set at 200.

@Service
public class MyService{

    public String execute(){
        try {
            //simulate blocking 
            Thread.sleep(3000);
        } catch (InterruptedException e) {}
        return "OK";      
    }
}

@RestController
public class MyController {

    @Autowired
    private MyService service;

    @RequestMapping("/test")
    public String test(){
        return service.execute();
    }
}

Scenario 2: I have a spring boot web app deployed to tomcat with tomcat min and max threads both set to 100

@Service
public class MyService{

    public String execute(){
        try {
            //simulate blocking 
            Thread.sleep(3000);
        } catch (InterruptedException e) {}
        return "OK";      
    }
}

@RestController
public class MyController {

    @Autowired
    private MyService service;

    private ExecutorService executorService = Executors.newFixedThreadPool(100);

    @RequestMapping("/test")
    public DeferredResult<String> test(){
        DeferredResult<String> deferredResult = new DeferredResult<>();
        CompletableFuture.supplyAsync(service::execute, executorService).
            whenCompleteAsync((result, throwable) -> deferredResult.setResult(result));    
        return deferredResult;      
    }
}

In each scenarios, the total number of threads is 200.

But I don't see how scenario 2 will perform any better:

In scenario 1, if 400 requests come in at the same time, the first 200 will be served by the 200 http threads and the next 200 will have to wait 3 seconds (plus a bit) until one of the threads becomes available again.

So the throughput was 400 requests per 6 seconds = 66.6 requests per second.

Average response time was (200 * 3 + 200 * 6)/(400) = 4.5 seconds

In scenario 2, if 400 requests come in at the same time. The first 100 will be served immediately by the 100 http threads, each of these threads will call the service, not wait for result, and then resume immediately, and become available to serve the next 100 requests. But now for the second 100 requests, when each of the http threads calls the service, that service is currently waiting 3 seconds (minus a bit) to finish processing the first 100 threads. so the next 100 get queued (in the executorservice's thread pool). So in almost no time at all, we've handled all 400 requests, but 100 are being processed in the service (waiting 3 seconds), while 300 are queued in executor service thread pool. 3 seconds later, first 100 are done, next 100 dequeued and processed and so on.

So the throughput is 400 requests in 12 seconds = 33.3 requests per second

Average response time was (100 * 3 + 100 * 6 + 100 * 9 + 100 * 12) / (400) = 7.5 seconds

Now, someone could argue, 'I can improve scenario 2 by increasing the number of threads in the executor service thread pool', to which I could reply, 'Fine, then I get to increase number of threads in tomcat pool in scenario 1 by the same amount'

Answer 1

To see the advantages of async in this scenario, you would need to make your Service also async. Instead of doing a Sleep that ties up the thread, it returns immediately, after scheduling something to run on completion three seconds later. In this case, all the requests would be complete in little more than three seconds; your throughput would be 133 requests per second, and average response time would be three seconds. And you would have essentially the same response time if you tuned the thread count way down.

The point of async is that your threads that are idle waiting for I/O are immediately free to do something else, so you don't have to use as many threads, which are an expensive resource, to meet your workload.

Answer 2

You've got a very synthetic situation in your question.

Let's say you've got 10 threads in both (10 HTTP threads and 5+5 threads for the async version), and your program involves more than calling a method that sleeps. However, 80% of your requests do involve an operation taking 3 seconds (let's say a database query).

Now in both cases it has happened that you've managed to get simultaneously all of your threads to call the blocking method. So far, there's no huge difference. If another call comes to the blocking method, it will have to wait.

Now, suddenly you get a request for a different operation, let's say login. Login is simple and just checks for a row in the db. In the first case it'll have to wait for 3 seconds, because there are no available HTTP threads to service it. In the second case, you've got a totally unrelated threadpool that's full, but since you're not using it for login, you get your login request serviced immediately.

Okay, so why not create a size 1000 threadpool without using DeferredResult ? Threads are expensive. You don't want to run into a situation where you've managed to get 1000 threads that are performing some expensive task, your CPU is at 100% and instead of 3 seconds, the runtime becomes 30 seconds for each request. Your server chokes and throughput goes to 0.

This also applies to connection pools. Less is more.