Java, MalformedChunkCodingException

Question

I have an Android Application that gives me this exception:

org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk

The exception is thrown from this method: (Purpose is to write out the response received from the server to a file.)

public static void getResponseBodyForServerData(
    final HttpEntity entity) throws IOException, ParseException {

    if (entity == null) {
        throw new IllegalArgumentException("HTTP entity may not be null");
    }
    InputStream instream = entity.getContent();

    if (instream == null) {
        return;
    }
    File file = new File(Environment.getExternalStorageDirectory() + 
        "/foo/Response.txt");

    if (!file.exists()) {
        file.createNewFile();
    }
    OutputStream out = new FileOutputStream(file);
    byte buf[] = new byte[1024];
    int len;
    while ((len = instream.read(buf)) > 0)
        out.write(buf, 0, len);
    out.flush();
    out.close();
}

So then I modified the above code to:

public static void getResponseBodyForServerData(
    final HttpEntity entity) throws IOException, ParseException {

    if (entity == null) {
        throw new IllegalArgumentException("HTTP entity may not be null");
    }
    InputStream instream = entity.getContent();
    InputStreamReader inputStreamReader = new InputStreamReader(
        instream, "UNICODE") ;

    if (instream == null) {
        return;
    }
    File file = new File(Environment.getExternalStorageDirectory() + 
        "/foo/Response.txt");
    if (!file.exists()) {
        file.createNewFile();
    }

    BufferedReader bufferedReader = new BufferedReader(inputStreamReader) ;
    BufferedWriter bufferedWriter = new BufferedWriter(new OutputStreamWriter(
        new FileOutputStream(file),"UNICODE")) ;

    String singleLine = null ;

    while((singleLine = bufferedReader.readLine()) != null) {
        bufferedWriter.write(singleLine) ;
    }

    bufferedWriter.flush() ;
    bufferedWriter.close() ;
}

And then the code works, What causes that error and what is wrong with the original code?

Answer 1

I see that the question is rather old, nevertheless it showed up, when I searched for MalformedChunkCodingException. So here are some hints what's going on.

I can only guess what your problem was since you don't tell us in which line the exception occured and which version of org.apache.http.HttpEntity you use.

First MalformedChunkCodingException: CRLF expected at end of chunk really is the problem. It occures when you have a chunked http transfer and at least one \\r\\n is missing in that transfer.

There are two possible errors. First the server does not send them or the client looses them.

To check if the server is ok check if the request response is ok. For example you can use curl to see the response in hex:

curl -ivs --raw http://host:port/somePath/toResource --trace /dev/stdout

Here is some sample output:

00a0: 30 31 22 7d 5d 0d 0a 30 0d 0a 0d 0a             01"}]..0....

See the \\r\\n as 0d 0a in the output? That means the server actually sends a correct message.

So you probably loose some bytes while reading the response.

I used a debugger and set a breakpoint for MalformedChunkCodingException (Java Exception Breakpoint in Eclipse). I looked in the stack where the incoming response was processed the first time and really there was only half of the response in the buffer.

A chunked message is a streaming message. So it may be that you close the socket before the InputStream has read the complete message . This for example happened in the jersey apache connector .

They had this code:

return new FilterInputStream(inputStream) {
    @Override
    public void close() throws IOException {
        response.close();
        super.close();
    }

which failed:

JulijanasJezov on 18 Oct 2016

This change does not work with the org.apache.httpcomponents: httpclient: 4.5.1+ when using chunked transfer encoding. This is because they have changed the way they close the connection.

In this case when response.close() is called the connection is closed and the input stream's buffer is cleared, then super.close() function is not able to close the ChunkedInputStream and results in org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected.

The reason of the exception occurring is that upon close of ChunkedInputStream, the close() function reads the remainder of the chunked message and it fails because the buffer of the stream has been cleared when the connection has been closed in response.close() call. Reference: https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/ChunkedInputStream.html#close()

The fix to this problem is putting super.close() line of code before the response.close(). This allows for the stream to be closed correctly before its buffer has been cleared.

So my problem was solved when I found out that in our project we used incompatible versions of jersey-apache-connector (old) and http-client (new) in our maven pom. (Depenency Hierachy said: httpclient:4.5.3 (managed from 4.5) (ommitted for conflict with 4.5.3) )