Java POST multipart/form-data containing json and bz2 compressed file parts

Question

So I am running into an issue where I need to POST a bzip2 file through a java app. I do not have an issue with doing multi-part posts or using regular files (See the third example located on http://zewaren.net/site/?q=node/123 )

However, the above example assumes that you are sending the file content as a string, where I would just to send compressed content (byte array).

    public void postSegments(HttpsURLConnection connection, File segment) throws UnsupportedEncodingException, IOException {
    PrintWriter OutputStream = null;
    try {
        OutputStream = new PrintWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"), true);

        OutputStream.append(BOUNDARY_SPLIT).append(BOUNDARY).append("\r\n");
        OutputStream.append("Content-Disposition: form-data; name=\"").append("metadata").append("\"").append("\r\n");
        OutputStream.append("Content-Type: ").append("application/json; charset=UTF-8").append("\r\n").append("\r\n");
        OutputStream.append(META_JSON).append("\r\n");

        OutputStream.append(BOUNDARY_SPLIT).append(BOUNDARY).append("\r\n");
        OutputStream.append("Content-Disposition: form-data; name=\"data\"; filename=\"NAME\"").append("\r\n");
        OutputStream.append("Content-Type: ").append("application/octet-stream; charset=UTF-8").append("\r\n").append("\r\n");
        OutputStream.append(fileService.readFile(segment));
        OutputStream.append(BOUNDARY_SPLIT).append(BOUNDARY).append(BOUNDARY_SPLIT).append("\r\n");
    }
    finally {
        if (OutputStream != null) {
          OutputStream.close();
        }
    }

In the code above I am getting an error when attempting to read in the segment file using a generic file scanner (Which does not work as segments is a bzip2 file)

I have been looking for a good wrapper lib to easily handle sending these multipart requests that handles large/zipped files well, but I have yet to have success.

Thank you all ahead of time.

Answer 1

So I had no success in posting files using httpsUrlConnection. I could not locate why, I think it was an issue with the headers that I was declaring.

However, I did manage to write some basic handlers using Jersey, as well as apache components.

The jersey client has its own build in Oauth, while the apache client uses an oauth-signpost plugin.

NOTE: These examples can post multipart/form-data consisting of files and json. Other inputs would be fairly easy to implement. It is also using Oauth, not Oauth2 which would require different libs and handles much differently (Ie token refreshes)

Below is the Jersey implementation. I would not recommend using this on large files as you would be storing the entire file in memory...You could break this up into chunks and send it, but I did not do that as I found the apache version to be much more suited to my needs. Regardless, here is the code:

import java.io.ByteArrayInputStream;
import java.io.File;

import javax.ws.rs.core.MediaType;
import javax.ws.rs.ext.Providers;

import org.apache.log4j.Logger;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.api.client.config.ClientConfig;
import com.sun.jersey.api.client.config.DefaultClientConfig;
import com.sun.jersey.core.header.FormDataContentDisposition;
import com.sun.jersey.multipart.FormDataBodyPart;
import com.sun.jersey.multipart.FormDataMultiPart;
import com.sun.jersey.multipart.impl.MultiPartWriter;
import com.sun.jersey.oauth.client.OAuthClientFilter;
import com.sun.jersey.oauth.signature.OAuthParameters;
import com.sun.jersey.oauth.signature.OAuthSecrets;

public class OauthRequestJersey {

private static final Logger log = Logger.getLogger(OauthRequestJersey.class);

private Client client = null;

private FileServiceUtil fileService = new FileServiceUtil();

private FormDataMultiPart multiPart;

private WebResource resource;

/*
 * Oauth Request Wrapper constructor
 */
public OauthRequestJersey(String url, String oAuthConsumerKey, String oAuthConsumerSecret){
    init();
    Providers providers = this.client.getProviders();
    OAuthSecrets secrets = new OAuthSecrets().consumerSecret(oAuthConsumerSecret);
    OAuthParameters params = new OAuthParameters().consumerKey(oAuthConsumerKey).signatureMethod("HMAC-SHA1");
    OAuthClientFilter filter = new OAuthClientFilter(providers,params,secrets);

    resource = client.resource(url);
    resource.addFilter(filter);
}

private void init(){
    ClientConfig config = new DefaultClientConfig();
    config.getClasses().add(MultiPartWriter.class);
    multiPart = new FormDataMultiPart();
    client = com.sun.jersey.api.client.Client.create(config);       
}

public void addJsonPart(String name, String json){
    multiPart.field(name, json, MediaType.APPLICATION_JSON_TYPE);
}

public void addFilePart(String name, File file){
    byte[] data = fileService.fileToByteArray(file);
    FormDataBodyPart fdbp = new FormDataBodyPart(
            FormDataContentDisposition.name(name)
                    .fileName(file.getName()).build(),
                     new ByteArrayInputStream(data),
                     MediaType.APPLICATION_OCTET_STREAM_TYPE);

    multiPart.bodyPart(fdbp);
}

public ClientResponse finish(){
    ClientResponse response = null;
    response =  resource.type(MediaType.MULTIPART_FORM_DATA)
            .post(ClientResponse.class,multiPart);
    log.debug("Response for the call " + response);
    log.debug("Response for the call status " + response.getStatus());
    return response;
}

}

Maven for jersey:

    <dependency>
    <groupId>com.sun.jersey.contribs</groupId>
    <artifactId>jersey-multipart</artifactId>
    <version>1.18.1</version>
</dependency>
<dependency>
    <groupId>com.sun.jersey</groupId>
    <artifactId>jersey-client</artifactId>
    <version>1.18.1</version>
</dependency>
<dependency>
    <groupId>com.sun.jersey.oauth</groupId>
    <artifactId>oauth-signature</artifactId>
    <version>1.1.1-ea</version>
</dependency>
<dependency>
    <groupId>com.sun.jersey.oauth</groupId>
    <artifactId>oauth-client</artifactId>
    <version>1.1.1-ea</version>
</dependency>

Below is my apache implementation. This is my preferred method as it handles file streaming on its own and abstracts headers instead of having to manually craft them yourself! You will notice I added in some logic to handle get requests as well.

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import oauth.signpost.OAuthConsumer;
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
import oauth.signpost.exception.OAuthCommunicationException;
import oauth.signpost.exception.OAuthExpectationFailedException;
import oauth.signpost.exception.OAuthMessageSignerException;

import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

import org.apache.log4j.Logger;

public class OauthRequestApache {
private static final Logger log = Logger.getLogger(OauthRequestApache.class);

private static CloseableHttpClient client;

private static boolean isPost;

private static HttpPost postRequest;

private static HttpGet getRequest;

private static MultipartEntityBuilder builder;

public OauthRequestApache(String requestURL, String consumerKey, String consumerSecret, boolean post) throws IOException, OAuthMessageSignerException, OAuthExpectationFailedException, OAuthCommunicationException{
    isPost = post;
    OAuthConsumer consumer = new CommonsHttpOAuthConsumer(consumerKey,consumerSecret);

    if(isPost){
        postRequest = new HttpPost(requestURL);
        builder = MultipartEntityBuilder.create();
        consumer.sign(postRequest);
    }else{
        getRequest = new HttpGet(requestURL);
        consumer.sign(getRequest);
    }
    client = HttpClients.createDefault();
}

public void addJsonPart(String name, String value){
    builder.addTextBody(name, value, ContentType.APPLICATION_JSON);
}

public void addFilePart(String name, File file){
    builder.addBinaryBody(name, file);
}

public List<String> finish() throws ClientProtocolException, IOException{
    HttpResponse response;
    List<String> status = new ArrayList<String>();
    ResponseHandler<String> handler = new BasicResponseHandler();
    if(isPost){
        HttpEntity multipart = builder.build();
        postRequest.setEntity(multipart);
         response = client.execute(postRequest);
    }else{
        response = client.execute(getRequest);
    }
    String body = handler.handleResponse(response);
    status.add(Integer.toString(response.getStatusLine().getStatusCode()));
    status.add(body);
    client.close(); 
    return status;
}
}

maven for apache

    <dependency>
    <groupId>oauth.signpost</groupId>
    <artifactId>signpost-core</artifactId>
</dependency>
<dependency>
    <groupId>oauth.signpost</groupId>
    <artifactId>signpost-commonshttp4</artifactId>
    <version>1.2.1.1</version>
</dependency>

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-compress</artifactId>
    <version>1.9</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.4</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.3</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpmime</artifactId>
    <version>4.3.6</version>
</dependency>