Most efficient way to deal with Large Amount of HttpRequests

Question

I have a fairly straight forward question, I'm going to be doing a lot of HTTP Requests to get an XML payload and I need to parse the payload and make some changes to a database.

I'm going to be doing these requests pretty frequently so any optimization will be good.

I'm wondering if there is a more efficient way to do the requests as that will take the most amount of time. I'm writing this in Java, I use the SAXParser to parse the XML payload for the data that I need. I'm using the HttpURLConnection class for Http Requests and I just run a thread synchronized and wait for each request. I've tried concurrently running the requests and I run into some issue's. The computer I will be doing this on is very capable so it's not like I will be running into memory or processing issues. So my questions are:

1. Would it be more efficient to take a hash of the XML payload and store it in the DB or parse the XML and check the data manually?
2. Is there a more efficient library for HttpRequests than HttpURLConnection?
3. Can any optimizations be made with threading the requests?

Any insight is appreciated.

Answer 1

Would it be more efficient to take a hash of the XML payload and store it in the DB or parse the XML and check the data manually?

Computing a hash for the XML payload will almost certainly be faster than parsing it. Thats a good way of detecting if you already have a payload that does not need to be re-parsed.

Is there a more efficient library for HttpRequests than HttpURLConnection?

Sure, use Apache HttpComponents . There are several ways to tune it for performance, as highlighted in this legacy but still relevant document .

Can any optimizations be made with threading the requests?

As with all things thread related, this will depend on several factors including your hardware configuration (particularly your network hardware). Theoretically you should gain some benefit from multithreading. I would not make it the number one priority for tuning.