Embedded Jetty timeout under load

Question

I have an akka (Java) application with camel-jetty consumer. Under some minimum load (about 10 TPS), our client starts seeing HTTP 503 error. I tried to reproduce the problem in our lab, and it seems jetty can't handle overlapping HTTP requests. Below is the output from apache bench (ab):

ab sends 10 requests using one single thread (ie one request at a time)

ab -n 10 -c 1 -p bad.txt http://192.168.20.103:8899/pim

Benchmarking 192.168.20.103 (be patient).....done


Server Software:        Jetty(8.1.16.v20140903)
Server Hostname:        192.168.20.103
Server Port:            8899

Document Path:          /pim
Document Length:        33 bytes

Concurrency Level:      1
Time taken for tests:   0.61265 seconds
Complete requests:      10
Failed requests:        0

Requests per second:    163.23 [#/sec] (mean)
Time per request:       6.126 [ms] (mean)
Time per request:       6.126 [ms] (mean, across all concurrent requests)

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.0      1       2
Processing:     3    4   1.8      5       7
Waiting:        2    4   1.8      5       7
Total:          3    5   1.9      6       8

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      8
  90%      8
  95%      8
  98%      8
  99%      8
  100%      8 (longest request)

ab sends 10 requests using two threads (up to 2 requests at the same time):

ab -n 10 -c 2 -p bad.txt http://192.168.20.103:8899/pim


Benchmarking 192.168.20.103 (be patient).....done


Server Software:        Jetty(8.1.16.v20140903)
Server Hostname:        192.168.20.103
Server Port:            8899

Document Path:          /pim
Document Length:        33 bytes

Concurrency Level:      2
Time taken for tests:   30.24549 seconds
Complete requests:      10
Failed requests:        1
   (Connect: 0, Length: 1, Exceptions: 0)

// obmited for clarity


Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.9      1       2
Processing:     3 3005 9492.9      4   30023
Waiting:        2 3005 9492.7      3   30022
Total:          3 3006 9493.0      5   30024


Percentage of the requests served within a certain time (ms)
  50%      5
  66%      5
  75%      7
  80%      7
  90%  30024
  95%  30024
  98%  30024
  99%  30024
  100%  30024 (longest request)

I don't believe jetty is this bad. Hopefully, it's just a configuration issue. This is the setting for my camel consumer URI:

"jetty:http://0.0.0.0:8899/pim?replyTimeout=70000&autoAck=false"

I am using akka 2.3.12 and camel-jetty 2.15.2

Answer 1

Jetty is certain not that bad and should be able to handle 10s of thousands of connections with many thousands of TPS.

Hard to diagnose from what you have said, other than Jetty does not send 503's when it is under load.... unless perhaps if the Denial of Service protection filter is deployed? (and ab would look like a DOS attack.... which it basically is and is not a great load generator for benchmarking).

So you need to track down who/what is sending that 503 and why.

Answer 2

It was my bad code: the sender (client) info was overwritten with overlapping requests. The 503 error message was sent due to Jetty continuation timeout.