How to disable buffering in Java HttpURLConnection for multi-part form post?

Question

(This is for a signed applet and I have decided against HTTPClient to keep my jar very small)

I am using HttpURLConnection to successfully upload a file from the user to a server using multi-part form post.

The problem is that HttpURLConnection is caching the data -- before sending it. So when I am reading from the file and writing to Outputstream, it is merely buffering the data -- and therefore my progress bar, that shows the upload status, is completely wrong. Howevere please note that the form post code works and the file does get uploaded correctly with return code of 200.

So how do I ensure that HttpURLConnection does not cache the data that I am sending to the server?

Here is my source:

public UploadResponse send(String formPostUrlStr,String fileFieldName,File targetFile, Map<String, String> valuesMap, UploadStatusListener uploadStatusListener) throws Exception{


    String sendStr=getBoundaryMessage(Boundary,  valuesMap, fileFieldName, targetFile.getName(), valuesMap.get("content-type") );//"image/png") ;

    System.out.println(" multi-part start \n "+ sendStr+ " multi-part end \n");

    String lenstr=Long.toString((long)(sendStr.length()*2)+ targetFile.length());
    System.out.println("Content-Length"+ lenstr);
                        //Content-Length

    URL url= new URL(formPostUrlStr);
    long startTime= System.currentTimeMillis();
    HttpURLConnection s3Connection = (HttpURLConnection) url.openConnection();

    System.out.println("opened url to "+ formPostUrlStr +", connection ok ..");
    s3Connection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
            + Boundary);
    s3Connection.setRequestProperty("content-length",  lenstr);




    s3Connection.setDoOutput(true);
    s3Connection.setDoInput(true);
    s3Connection.setUseCaches(false);
     s3Connection.setInstanceFollowRedirects(true);
    s3Connection.setAllowUserInteraction(true);
    s3Connection.setRequestProperty("User-Agent", "Mozilla/4.5");


    if (uploadStatusListener != null) {
        uploadStatusListener.statusUpdate(targetFile.length(), 0);
    }


    String debugStr= s3Connection.toString();
    System.out.println("conmnection "+ debugStr);


    DataOutputStream httpOut = new DataOutputStream(s3Connection.getOutputStream());

    System.out.println("opened DataOutputStream ok ..");




    httpOut.write(sendStr.getBytes());


    //httpOut.flush();

  System.out.println("httpOut.flush 1 ok ..");
    FileInputStream uploadFileReader = new FileInputStream(targetFile);
    long totalBytes = uploadFileReader.available();
    if (uploadStatusListener != null) {
        uploadStatusListener.statusUpdate(totalBytes, 0);
    }
    System.out.println(" uploading file with size  "+ uploadFileReader.available());
    int bufSize = 102400;
    long availableBytesToRead;
    long totalSent = 0;
    while ((availableBytesToRead = uploadFileReader.available()) > 0) {
        byte[] bufferBytesRead;
        bufferBytesRead = availableBytesToRead >= bufSize ? new byte[bufSize]
                : new byte[(int)availableBytesToRead];
        int count = uploadFileReader.read(bufferBytesRead);
        try{
            httpOut.write(bufferBytesRead);
            totalSent += ((long) count);
            System.out.println(" wrote bytes = "+count+ ", total sent = "+ totalSent +", pendingSize"+ (availableBytesToRead-count) );
        }
        catch(IOException ioe){
            System.out.println(" io exceotion e"+ ioe.getMessage());
            throw ioe;
        }

        //httpOut.flush();
        if (uploadStatusListener != null) {
            uploadStatusListener.statusUpdate(totalBytes, totalSent);
        }

    }
    // FILE DATA END 
    httpOut.write(("--" + Boundary + "--\r\n").getBytes());



    // form end     
    httpOut.write(("--" + Boundary + "--\r\n").getBytes());

    httpOut.flush();
    httpOut.close();

    long endTime= System.currentTimeMillis();

    System.out.println("Completed Writing Data to S3 Connection in "+ (endTime-startTime)+"ms.,now waiting for rsponse code ");
    int code=s3Connection.getResponseCode();
    long endTime2= System.currentTimeMillis();
    System.out.println("Completed Sendind Data to S3 in "+ (endTime2-startTime)+ "ms., rsponse code time "+ (endTime2-endTime)+"ms. ");


    UploadResponse uploadResponse = new UploadResponse();

    uploadResponse.setCode(code);

    System.out.println(" response code : " + code);

    StringBuilder response = new StringBuilder();
    byte[] respBuffer = new byte[4096];


    if (code > 300) {
        if (code == 404) {
            throw new Exception("Error 404");
        }
        BufferedReader err = new BufferedReader(
                new InputStreamReader(s3Connection.getErrorStream()));
        String ret;
        StringBuffer buff = new StringBuffer();
        while ((ret = err.readLine()) != null) {
            buff.append(ret);
        }
        uploadResponse.setMessage(buff.toString());
        System.out.println(" error :"+ buff.toString());
        err.close();


    } else {
        BufferedReader inp = new BufferedReader(
                new InputStreamReader(s3Connection.getInputStream()));
        StringBuffer buff = new StringBuffer();
        String ret;
        while ((ret = inp.readLine()) != null) {
            buff.append(ret);
        }
        inp.close();
        uploadResponse.setMessage(buff.toString());
        if(buff.toString().contains("fail"))
            throw new Exception("Upload failed");
    }        

    System.out.println(response.toString());

    return uploadResponse;        

}

}

Answer 1

I have the same problem. I didn't find any other solution than writing my HTTP request on a raw Socket. Did you find a better workaround?

EDIT: I just did: we just have to use obj.setFixedLengthStreamingMode(12345) on the HttpURLConnection object obtained from url.openConnection(), where 12345 is the length of POST request body.

Answer 2

As a complementation for the answer that @Antares gave, there is another method setChunkedStreamingMode that is used when you don't know the content size in advance. So when you do a POST request, call that method on the connection:

HttpURLConnection connection = ...
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setChunkedStreamingMode(0);
connection.connect();
... connection.getOutputStream();

This will avoid the OutputStream to buffer the entire content before start to send.