Jetty Async not working as anticipated, experiencing worse performance than sync

Question

Thanks in advance for any pointers or help.

Basically I was expecting the async version of this to perform far better than the sync version, but instead the sync version performs equivalent or somehow better.

Am I doing something wrong, what gives? I tried without Javalin in case something in the framework was creating a problem, seemed to give similar results. I did try this with just Netty (too long to post the code) and I experienced similar results also.

I wrote the following code: (javalin-3.12.0 and jetty-9.4.31.v20200723)

import io.javalin.Javalin;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.util.thread.QueuedThreadPool;
import java.io.IOException;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

public class AsyncTest {
    static ScheduledThreadPoolExecutor scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(5000);
    public static void main(String[] args) {
        var jav = Javalin.create();
        jav.config.server(() -> new Server(new QueuedThreadPool(5000, 500, 120_000)));
        Javalin app = jav.start(8080);

        app.get("/async-delay", ctx -> {
            var async = ctx.req.startAsync();
            scheduledThreadPoolExecutor.schedule(() -> {
                try {
                    ctx.res.getOutputStream().println("ok");
                } catch (IOException e) {
                    e.printStackTrace();
                }
                async.complete();

            }, 100, TimeUnit.MILLISECONDS);
        });

        app.get("/delay", ctx -> {
            Thread.sleep(100);
            ctx.result("ok");
        });

        app.get("/no-delay", ctx -> {
            ctx.result("ok");
        });
    }
}

And got the following results:

➜  ~ wrk2 -t16 -c300 -d5s -R3000 http://localhost:8080/delay
Running 5s test @ http://localhost:8080/delay
  16 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   331.36ms  138.72ms 626.18ms   57.34%
    Req/Sec        nan       nan   0.00      0.00%
  10854 requests in 5.00s, 1.24MB read
  Socket errors: connect 53, read 0, write 0, timeout 106
Requests/sec:   2170.40
Transfer/sec:    254.34KB
➜  ~ wrk2 -t16 -c300 -d5s -R3000 http://localhost:8080/async-delay
Running 5s test @ http://localhost:8080/async-delay
  16 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   285.84ms  120.75ms 522.50ms   56.27%
    Req/Sec        nan       nan   0.00      0.00%
  11060 requests in 6.10s, 1.29MB read
  Socket errors: connect 53, read 0, write 0, timeout 124
Requests/sec:   1814.16
Transfer/sec:    216.14KB
➜  ~ wrk2 -t16 -c16 -d5s -R70000 http://localhost:8080/no-delay
Running 5s test @ http://localhost:8080/no-delay
  16 threads and 16 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.51ms    3.12ms  21.95ms   88.36%
    Req/Sec        nan       nan   0.00      0.00%
  349824 requests in 5.00s, 40.03MB read
Requests/sec:  69995.44
Transfer/sec:      8.01MB

Answer 1

Since Jetty 9+ is 100% async from the get go, this lack of difference makes sense. (In fact, in Jetty 9+ there is extra work done to pretend to be synchronous when using synchronous APIs like InputStream.read() or OutputStream.write() )

Also, your load testing workloads are not realistic.

• You want many more client machines to do the testing with. There is no single software client alone is capable of stressing a Jetty server. You'll hit system resource limits well before you hit any kind of Jetty serving limits.
  • At least a 4 to 1 ratio (we test with a 8 to 1 ratio) of client machines to server machines to generating enough load to stress Jetty.
• You want many concurrent connections to the server. (think 40,000+)
  • Or you want HTTP/2 in the picture (which also stresses out the server resources in it's own unique ways)
• You want lots of data returned (something that would take multiple network buffers to return).
• You want to sprinkle in a few client connections that are slow to read as well (which on a synchronous server can impact the rest of the connections that are not slow by simply consuming too many resources)

Answer 2

Yep, Jaokim called it, wrk is the bottleneck here. If I run them in parallel as suggested above it 4x the number of rps.

➜  ~ ➜  ~ wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay & ; wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay & ; wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay & ; wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay & ;
[1] 2779
[2] 2780
[3] 2781
[4] 2782
Running 5s test @ http://localhost:8080/async-delay
  16 threads and 500 connections
Running 5s test @ http://localhost:8080/async-delay
  16 threads and 500 connections
Running 5s test @ http://localhost:8080/async-delay
  16 threads and 500 connections
Running 5s test @ http://localhost:8080/async-delay
  16 threads and 500 connections
➜  ~   Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   104.09ms   13.99ms 337.66ms   97.43%
    Req/Sec        nan       nan   0.00      0.00%
  7066 requests in 5.06s, 841.85KB read
  Socket errors: connect 261, read 14, write 1, timeout 522
Requests/sec:   1395.35
Transfer/sec:    166.24KB
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   103.84ms   12.70ms 239.23ms   97.48%
    Req/Sec        nan       nan   0.00      0.00%
  7066 requests in 5.06s, 841.85KB read
  Socket errors: connect 261, read 9, write 2, timeout 522
Requests/sec:   1395.56
Transfer/sec:    166.27KB

[1]    2779 done       wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay
[2]    2780 done       wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay
➜  ~ ➜  ~   Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   103.62ms   12.48ms 243.58ms   97.67%
    Req/Sec        nan       nan   0.00      0.00%
  7064 requests in 6.16s, 841.61KB read
  Socket errors: connect 261, read 13, write 2, timeout 584
Requests/sec:   1147.51
Transfer/sec:    136.71KB
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   103.50ms   12.94ms 339.46ms   97.83%
    Req/Sec        nan       nan   0.00      0.00%
  7055 requests in 6.16s, 840.54KB read
  Socket errors: connect 261, read 6, write 2, timeout 646
Requests/sec:   1145.42
Transfer/sec:    136.47KB

[3]  - 2781 done       wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay
[4]  + 2782 done       wrk2 -t16 -c500 -d5s -R3000 http://localhost:8080/async-delay
➜  ~ ➜  ~