Upload 3 GB document using apache HttpClient 4.1.2

Question

We are using HttpClient 4.1.2. When we try to upload a document to the server, which is working good if we try to upload a document of size less than 2 GB, If I upload more than 2 GB document then I see the error below..

java.net.SocketException: Connection reset by peer: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:297)
    at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:286)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:743)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:731)
    at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:153)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:114)
    at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:90)
    at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1859)

Please find the code below:

try {
    // Requires a multi-part post
    MultipartEntity requestEntity = new MultipartEntity();
    post.setEntity(requestEntity);

    requestEntity.addPart(MIME_TYPE_NAME, new StringBody(mimeType));

    requestEntity.addPart(USER_ID_PARAMETER, new StringBody(loginUser
            .getUserid().toLowerCase()));
    requestEntity.addPart(USER_PASSWORD_PARAMETER, new StringBody(
            loginUser.getPassword()));

    requestEntity.addPart(DOCUMENT_PART_NAME,
            new ProgressReportingFileBody(filePath, uploadListener));

    post.getParams().setBooleanParameter(
            CoreProtocolPNames.USE_EXPECT_CONTINUE, true);

    // Find out what happened with the request
    HttpContext context = new BasicHttpContext();
    // Add AuthCache to the execution context
    context.setAttribute(ClientContext.AUTH_CACHE, authCache);
    // Perform the POST
    setCurrentRequest(post);
    HttpResponse response = httpClient.execute(targetHost, post,
            context);
    // Get the response back
    message = response.getEntity();

    // Ensure the middleware returned an acceptable response
    checkError(response, context);
    checkInterrupted(post);

    Header documentId = response.getFirstHeader("DOCUMENT_ID");

    oDoc.sMimeType = mimeType;

} catch (ClientProtocolException ex) {
    throw new IwException(-3, "HTTP Error: " + ex.getMessage(),
            ex.toString());
} catch (IOException ex) {
    checkInterrupted(post);
    log.error("Failed to Upload Document", ex);
    throw new IwException(-3, "Upload Document Error: "
            + ex.getMessage(), ex.toString());
} finally {
    try {
        if (message != null) {
            // Clean up to allow another HTTP client call
            EntityUtils.consume(message);
        }
    } catch (IOException ex) {
        log.error("Failed to Close HTTP Connection", ex);
    }
}

Any ideas what it could be / would be great! Thanks for your time.

Answer 1

Upload in chunks, each having size of 256 Mb or about. Join on the sever side. On the programming level, this is easy to do. Also, uploading 3 Gb probably takes very long. In case of failure, it makes sense to have possibility to retry with smaller chunk.