如何限制HttpClient响应长度

Question

I'm using htmlunit with httpclient . How can I limit the response body length of httpclient to say 1MB ?

Answer 1

The trick is easy. You have to take the InputStream,read from it and stop reading when the limit is exceeded.

      InputStream instream = method.getResponseBodyAsStream();

I have done an example tuning the apache example a bit.

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.params.HttpMethodParams;
public class HttpClientTutorial {

  private static String url = "http://www.apache.com";
  private static final int LIMIT = 1024*1024;//set to 1MB
  public static void main(String[] args) {
    // Create an instance of HttpClient.
    HttpClient client = new HttpClient();
    // Create a method instance.
    GetMethod method = new GetMethod(url);   
    // Provide custom retry handler is necessary
    method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, 
            new DefaultHttpMethodRetryHandler(3, false));
    try {
      // Execute the method.
      int statusCode = client.executeMethod(method);
      if (statusCode != HttpStatus.SC_OK) {
        System.err.println("Method failed: " + method.getStatusLine());
      }
      // Read the response body.
      //byte[] responseBody = method.getResponseBody();
      byte[] responseBody = null;    
      InputStream instream = method.getResponseBodyAsStream();
      if (instream != null) {
          long contentLength = method.getResponseContentLength();
          if (contentLength < Integer.MAX_VALUE) { //guard below cast from overflow
              ByteArrayOutputStream outstream = new ByteArrayOutputStream();
              byte[] buffer = new byte[1024];
              int len;
              int total = 0;
              while ((len = instream.read(buffer)) > 0 && total<LIMIT) {
                  outstream.write(buffer, 0, len);
                  total+= len;
              }
              responseBody = outstream.toByteArray();
              outstream.close();
              instream.close();
              System.out.println(new String(responseBody));
          }
      }
    } catch (HttpException e) {
      System.err.println("Fatal protocol violation: " + e.getMessage());
      e.printStackTrace();
    } catch (IOException e) {
      System.err.println("Fatal transport error: " + e.getMessage());
      e.printStackTrace();
    } finally {
      // Release the connection.
      method.releaseConnection();
    }  
  }
}

I hope this helps.

Answer 2

You could use BoundedInputStream from Apache Commons. This is a stream that will only supply bytes up to a certain length - if its position goes above that, it will stop.