Is it possible to peek HttpClient's org.apache.http.conn.EofSensorInputStream?

Question

I am trying to peek at an input stream contents from HttpClient, up to 64k bytes.

The stream comes from an HttpGet, nothing unusual about it:

HttpGet requestGet = new HttpGet(encodedUrl); HttpResponse httpResponse = httpClient.execute(requestGet); int status = httpResponse.getStatusLine().getStatusCode(); if (status == HttpStatus.SC_OK) { return httpResponse.getEntity().getContent(); }

The input stream it returns is of type org.apache.http.conn.EofSensorInputStream

Our use-case is such that we need to "peek" at the first (up to 64k) bytes of the input stream. I use an algorithm described here How do I peek at the first two bytes in an InputStream?

PushbackInputStream pis = new PushbackInputStream(inputStream, DEFAULT_PEEK_BUFFER_SIZE); byte [] peekBytes = new byte[DEFAULT_PEEK_BUFFER_SIZE]; int read = pis.read(peekBytes); if (read < DEFAULT_PEEK_BUFFER_SIZE) { byte[] trimmed = new byte[read]; System.arraycopy(peekBytes, 0, trimmed, 0, read); peekBytes = trimmed; } pis.unread(peekBytes);

When I use a ByteArrayInputStream, this works with no problem.

The Issue: When using the org.apache.http.conn.EofSensorInputStream I only get a small number of bytes at the beginning of the stream. usually around 400 bytes. When I expected up to 64k bytes.

I also tried using a BufferedInputStream where I read up to the first 64k bytes then call a .reset() but that doesn't work either. Same issue.

Why might this be? I do not think anything is closing the stream because if you call IOUtils.toString(inputStream) I do get all the content.

Answer 1

See InputStream#read(byte[] b,int off,int len) contract:

Reads up to len bytes of data from the input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read . The number of bytes actually read is returned as an integer

Instead of using this method, use IOUtils.read which reads until you get the number of bytes you requested in a loop.