Cause of Lag in Axis2 Web Service

Question

I have a Java SOAP web service initially designed in Axis 1 which isn't meeting my performance requirements.

The request I'm most concerned about is one used to add lots (millions of rows) of data to the database. On the client side, I'll just be looping through files, pushing this data up to my web service. Each row has three elements, so the request looks something like:

<SOAP Envelope/Header/Body>
<AddData>
  <Data>
    <FirstName>John</FirstName>
    <LastName>Smith</LastName>
    <Age>42</Age>
  </Data>
</AddData>
</SOAP Envelope/Body>

I'm finding the following performance trends:

• When I do one row per request, I can get around 720 rows per minute.
• When I encapsulate multiple rows into a single request, I can get up to 2,400 rows per minute (100 rows per request).

Unfortunately, that performance isn't going to meet our requirements, as we have hundreds millions of rows to insert (at 2,500 rows per minute, it would take about 2 months to load all the data in).

So I've been looking into the application to see where our bottleneck is. Each request of 100 rows is taking about 2.5 seconds (I've tried a few different servers and get similar results). I've found the following:

• Client-side overhead is negligible (from monitoring the performance of my own client and using SOAP UI)
• The database activity only accounts for about 10% (.2s) of the total time, so Hibernate caching, etc. won't help out much.
• The network overhead is negligible (<1ms ping time from client to server, getting >10MB/s throughput with each request sending <20KB).

So this leaves some 2 seconds unaccounted for. The only other piece of this puzzle that I can point a finger at is the overhead of deserializing the incoming requests on the server side. I noticed that Axis 2 claims speed improvements in this area, so I ported this function over the an Axis 2 web service but didn't get the speedup I was looking for (the overall time per request improved by about 10%).

Am I underestimating the amount of time needed to deserialize 100 of the elements described above? I can't imagine that that deserialization could possibly take ~2 seconds.

What can I do to optimize the performance of this web application and cut down on that 2 second overhead?

Thanks in advance!

========= The next day.... =========== The plot thickens...

At the recommendation of @millhouse, I investigated single row requests one a production server a bit more. I found that they could be suitably quick on good hardware. So I tried adding 1,000 rows using increments ranging from 1 (1,000 separate requests) to 1,000 (a single request).

• 1 row / Request - 14.5 seconds
• 3/req - 5.8s
• 5/req - 4.5s
• 6/req - 4.2s
• 7/req - 287s
• 25/req - 83s
• 100/req - 22.4s
• 1000/req - 4.4s

As you can see, the extra 2 second lag kicks in ay 7 rows per request (approximately 2 extra seconds per request when compared to 6 rows per request). I can reproduce this consistently. Larger numbers of requests all had similar overhead, but that became less noticeable when inserting 1,000 rows per request. Database time grew linearly and was still fairly negligible compared to the overall request time.

So I've found that I get best performance using either 6 rows per request, or thousands of rows per request.

Is there any reason why 7 would have such lower performance than 6 rows per request? The machine has 8 cores, and we have 10 connections in the session pool (ie I have no idea where the threshold of 6 is coming from).

Answer 1

I used Axis2 for a similar job about 5 years ago, but I'm afraid I can't offer any real "magic bullet" that will make it better. I recall our service performing at hundreds-per-second not seconds-per-hundred though.

I'd recommend either profiling your request-handling, or simply adding copious amounts of logging (possibly using one of the many stopwatch implementations around to give detailed timings) and seeing what's using the time. Does a request really take 2 seconds to get through the Axis layer to your code, or is it just accumulating through lots of smaller things?

If the processing for a single request in isolation is fast, but things get bogged down once you start loading the service up, investigate your app server's thread settings. I seem to recall having to break my processing into synchronous and asynchronous parts (ie the synchronous part doing the bare minimum to give a suitable response back to the client, and heavy-lifting being done in a thread from a pool), but that might not be appropriate for your situation.

Also make sure that construction of a new User object (or whatever it is) doesn't do anything too expensive (like grabbing a new ID, from a service, which wraps a DAO, which hits a slow database server, which runs a badly-written stored-procedure, which locks an entire table ;-) )