使用Java上传具有multipart / form-data的zip文件时丢失字节

Question

I'm having an issue with uploading a .zip file to a remote server in that some bytes are missing from the file after the upload. Upon redownloading the file, the .zip archive is unopenable, which leads me to believe that I need those bytes to perform the upload successfully.

I am using a multipart/form-data POST request to upload the file. The utility helper class I use to do this is given in the below code:

import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;

public class MultipartFormDataUtil {
    private final String boundary;
    private static final String lineReturn = "\r\n";
    private HttpURLConnection conn;
    private DataOutputStream dos;
    int bytesRead, bytesAvail, bufferSize;
    byte[] buffer;
    int maxBufferSize = 1*1024*1024;
    List<String> response;

    public MultipartFormDataUtil(String postUrl, LinkedHashMap<String, String> params, File file) throws IOException {
        boundary = "=-=" + System.currentTimeMillis() + "=-=";

        URL url = new URL(postUrl);
        conn = (HttpURLConnection) url.openConnection();
        conn.setDoInput(true);
        conn.setDoOutput(true);
        conn.setUseCaches(false);
        conn.setRequestMethod("POST");
        conn.setRequestProperty("Connection", "Keep-Alive");
        conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
        dos = new DataOutputStream(conn.getOutputStream());

        for (String key : params.keySet()) {
            addFormPart(key, params.get(key));
        }

        addFilePart(file);

        finish();
    }

    private void addFormPart(String name, String value) throws IOException {
        dos.writeBytes("--" + boundary + lineReturn);
        dos.writeBytes("Content-Disposition: form-data; name=\"" + name + "\"" + lineReturn);
        dos.writeBytes("Content-Type: text/plain" + lineReturn + lineReturn);
        dos.writeBytes(value + lineReturn);
        dos.flush();
    }

    private void addFilePart(File file) throws IOException {
        FileInputStream fileInputStream = new FileInputStream(file);

        dos.writeBytes("--" + boundary + lineReturn);
        dos.writeBytes("Content-Disposition: form-data; name=\"file\"; filename=\"" + file.getName() + "\"" + lineReturn);
        dos.writeBytes("Content-Type: " + URLConnection.guessContentTypeFromName(file.getName()) + lineReturn);
        dos.writeBytes("Content-Transfer-Encoding: binary" + lineReturn + lineReturn);

        bytesAvail = fileInputStream.available();
        bufferSize = Math.min(bytesAvail, maxBufferSize);
        buffer = new byte[bufferSize];

        bytesRead = fileInputStream.read(buffer, 0, bufferSize);

        while (bytesRead > 0) {
            dos.write(buffer, 0, bufferSize);
            bytesAvail = fileInputStream.available();
            bufferSize = Math.min(bytesAvail, maxBufferSize);
            bytesRead = fileInputStream.read(buffer, 0, bufferSize);
        }
        dos.flush();

        dos.writeBytes(lineReturn);
        dos.flush();
        fileInputStream.close();
    }

    private void finish() throws IOException {
        response = new ArrayList<String>();

        dos.writeBytes("--" + boundary + "--" + lineReturn);
        dos.flush();
        dos.close();

        BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
        String line;

        while ((line = reader.readLine()) != null) {
            response.add(line);
        }

        reader.close();
        conn.disconnect();
    }

    public List<String> getResponse() {
        return response;
    }

To give credit where credit is due, this utility is based off of examples from Peter's Notes and CodeJava.net . This util is called with the following code:

protected static void postFile(String url, LinkedHashMap<String, String> params, File file) throws Exception {
    try {
        MultipartFormDataUtil multipartRequest = new MultipartFormDataUtil(url, params, file);
        List<String> response = multipartRequest.getResponse();

        for (String line : response) {
            System.out.println(line);
        }

    } catch (IOException ioe) {
        log.warn("There was an error posting the file and form data", ioe);
    }
}

The upload url in this case is to an Amazon S3 bucket, which passes it on to the destination system. It is at this final destination that I can see that the process that is supposed to be running on the .zip file has failed (note: the process is run by a Rails app and gives the error "Error identifying package type: can't dup NilClass"). Upon downloading the file, I see that the file size is 3,110,416 bytes instead of 3,110,466 bytes. I can no longer extract the archive to see what is in it; the mac archive utility responds with "Error 2 - No such file or directory".

I lack the conceptual background in this area to get a feel for where in the process things may be going wrong. I am hoping that someone will be able to tell me that I made an error in the utility class, or let me know that something else is the problem.

Thank you for any insight you can provide, and let me know if I can post anything else that would be of help.

EDIT: Some additional information I gathered about different sizes of file uploads (in bytes):

Original----------Uploaded----------Difference

10,167,389______10,167,238______151

3,110,466_______3,110,416_______50

156,885_________156,885_________0

95,639,352______95,637,925______1,427

For the 3 files that had bytes missing following the upload, the % of total data lost was around (but not exactly) 0.0015% for each one, but not equal to each other.

Answer 1

Upon some further research, we found that the error did not have anything to do with the multipart/form-data utility as shown in this question. Instead, it had to do with our own file downloading client in here . We were not setting the FileTransfer client to download the file as binary, which is necessary for a .zip file.

Feel free to use the code included in the original question for your multipart/form-data purposes in java - it works great assuming there are no problems with your original file.