Slow reading of HTTP response using InputStreamReader and BufferedReader?

Question

I have the following code in Java that sends an HTTP request to a web server and read the response:

StringBuilder response = new StringBuilder(50000);
URL url2 = new URL(ServiceURL); 
connection = (HttpURLConnection)url2.openConnection();
connection.setRequestMethod("POST");

//... (some more connection settings) ...

BufferedWriter  wr = new BufferedWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));

wr.write(Request);
wr.flush ();
wr.close ();

InputStream is = connection.getInputStream();       
BufferedReader rd = new BufferedReader(new InputStreamReader(is));

int i = 0;
while ((i = rd.read()) > 0) {
    response.append((char)i);
}

It works for most cases, but I have a problem with one server that returns a rather large XML (something like 500KB; I guess this is pretty large for just a bunch of text..), where I keep getting a read timeout exception.

I believe it's not a network problem, because I've tried making the same request using curl and the response just arrived all right and pretty quick, something like two seconds.

When I look what's going on in the network (using wireshark to capture the packets), I noticed that the TCP receive window in my computer gets full at some point. The TCP stack sometimes survives this; I can see the server sending TCP keep-alive to keep the connection up, but in the end the TCP connection just breaks down.

Could it be that the reading part of the code (appending the received response character-by-character) is slowing my code down? Is there a more efficient way to read an HTTP response?

Answer 1

Reading character by character is quite slow, yes. Try reading chunks at a time into a buffer:

char[] buf = new char[2048];
int charsRead;
while((charsRead = rd.read(buf, 0, 2048)) > 0) {
    response.append(buf, 0, charsRead);
}

Answer 2

As Phil already said reading the stream byte by byte is kinda slow. I prefer using the readLine() method of BufferedReader :

StringBuilder response = new StringBuilder();
String line = "";
while((line = rd.readLine()) != null) {
    response.append(line + System.getProperty("line.separator");
}

If possible, I would consider using the Apache HTTP Client library. It is easy to use and very powerful in handling HTTP stuff. http://hc.apache.org/httpcomponents-client-ga/

You should also keep in mind to set the socket and connection timeouts. This way you can control how long a connection is kept open (alt least on you side of the connection).

And last but not least always close your HTTP connections in a finally block after you received the response, otherwise you may run into a too many open files problem.

Hope this heps ;)