How to properly upload (image) file to Google Cloud Storage using Java App Engine?

Question

I have a Google app engine instance, using java (sdk 1.9.7), and it is connected to Google Cloud Storage. I'm able to successfully take a request's input and output it to a file/object in my google cloud storage bucket. here's my code for my servlet:

public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
    // read the input stream
    byte[] buffer = new byte[1024];
    List<byte[]> allBytes = new LinkedList<byte[]>();
    InputStream reader = req.getInputStream();
    while(true) {
        int bytesRead = reader.read(buffer);
        if (bytesRead == -1) {
            break; // have a break up with the loop.
        } else if (bytesRead < 1024) {
            byte[] temp = Arrays.copyOf(buffer, bytesRead);
            allBytes.add(temp);
        } else {
            allBytes.add(buffer);
        }
    }

    // init the bucket access
    GcsService gcsService = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
    GcsFilename filename = new GcsFilename("my-bucket", "my-file");
    Builder fileOptionsBuilder = new GcsFileOptions.Builder();
    fileOptionsBuilder.mimeType("text/html"); // or "image/jpeg" for image files
    GcsFileOptions fileOptions = fileOptionsBuilder.build();
    GcsOutputChannel outputChannel = gcsService.createOrReplace(filename, fileOptions);

    // write file out
    BufferedOutputStream outStream = new BufferedOutputStream(Channels.newOutputStream(outputChannel));
    for (byte[] b : allBytes) {
        outStream.write(b);
    }
    outStream.close();
    outputChannel.close();
}

and when i do something like a curl POST command, this works perfectly if i just feed it data directly, like so:

curl --data "someContentToBeRead" http://myAppEngineProj.appspot.com/myServlet

and i can see the exactly string that i put in, "someContentToBeRead".

HOWEVER, when i put a file, like so:

curl -F file=@"picture.jpg" http://myAppEngineProj.appspot.com/myServlet

the file is completely corrupted. if i upload a text file, it has a line of crap in the beginning of the file, and a line of crap at the end, like:

------------------------------266cb0e18eba
Content-Disposition: form-data; name="file"; filename="blah.txt"
Content-Type: text/plain

hi how are you

------------------------------266cb0e18eba--

how do i tell cloud storage i want to store the data as file?

Answer 1

This is what worked for me

To upload, use

curl -F file=@"picture.jpg" http://myAppEngineProj.appspot.com/myServlet

And the servlet looks like

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.nio.channels.Channels;
import java.util.Enumeration;
import java.util.logging.Logger;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.fileupload.FileItemIterator;
import org.apache.commons.fileupload.FileItemStream;
import org.apache.commons.fileupload.servlet.ServletFileUpload;

import com.google.appengine.tools.cloudstorage.GcsFileOptions;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsOutputChannel;
import com.google.appengine.tools.cloudstorage.GcsService;
import com.google.appengine.tools.cloudstorage.GcsServiceFactory;
import com.google.appengine.tools.cloudstorage.RetryParams;

public class UploadServlet extends HttpServlet {

    private static final Logger log = Logger.getLogger(UploadServlet.class.getName());

    private final GcsService gcsService = GcsServiceFactory.createGcsService(new RetryParams.Builder()
    .initialRetryDelayMillis(10)
    .retryMaxAttempts(10)
    .totalRetryPeriodMillis(15000)
    .build());

    private String bucketName = "myBucketNameOnGoogleCloudStorage";

    /**Used below to determine the size of chucks to read in. Should be > 1kb and < 10MB */
      private static final int BUFFER_SIZE = 2 * 1024 * 1024;

    @SuppressWarnings("unchecked")
    @Override
    public void doPost(HttpServletRequest req, HttpServletResponse res)
            throws ServletException, IOException {

        String sctype = null, sfieldname, sname = null;
        ServletFileUpload upload;
        FileItemIterator iterator;
        FileItemStream item;
        InputStream stream = null;
        try {
            upload = new ServletFileUpload();
            res.setContentType("text/plain");

            iterator = upload.getItemIterator(req);
            while (iterator.hasNext()) {
                item = iterator.next();
                stream = item.openStream();

                if (item.isFormField()) {
                    log.warning("Got a form field: " + item.getFieldName());
                } else {
                    log.warning("Got an uploaded file: " + item.getFieldName() +
                            ", name = " + item.getName());

                    sfieldname = item.getFieldName();
                    sname = item.getName();

                    sctype = item.getContentType();

                    GcsFilename gcsfileName = new GcsFilename(bucketName, sname);

                    GcsFileOptions options = new GcsFileOptions.Builder()
                    .acl("public-read").mimeType(sctype).build();

                    GcsOutputChannel outputChannel =
                            gcsService.createOrReplace(gcsfileName, options);

                    copy(stream, Channels.newOutputStream(outputChannel));

                    res.sendRedirect("/");
                }
            }
        } catch (Exception ex) {
            throw new ServletException(ex);
        }
    }

    private void copy(InputStream input, OutputStream output) throws IOException {
        try {
          byte[] buffer = new byte[BUFFER_SIZE];
          int bytesRead = input.read(buffer);
          while (bytesRead != -1) {
            output.write(buffer, 0, bytesRead);
            bytesRead = input.read(buffer);
          }
        } finally {
          input.close();
          output.close();
        }
      }

}

References : Wilson Yeung's answer above and This Post

Although the other post has a limitation of upload size < 32 mb, that was not a problem for me. And this code also handles mime types automatically.

Answer 2

As far as I can tell, there is no problem with Google Cloud Storage or the APIs; the problem is earlier, in the reading of the content from HttpServletRequest.

The lines containing ------266cb0e18eba are actually part of the MIME encoding and marks the beginning and end of a part.

You can resolve the issue in one of two ways.

Option A: Keep the code the same, but change the way you upload data

Replace:

$ curl -F file=@"picture.jpg" http://myAppEngineProj.appspot.com/myServlet

With:

$ curl -X POST -d @"picture.jpg" http://myAppEngineProj.appspot.com/myServlet

Option B: Fix the Java code and continue using curl as you are using it

Replace:

java.io.InputStream is = request.getInputStream();

With:

javax.servlet.http.Part filePart = request.getPart("file");
java.io.InputStream is = filePart.getInputStream()

Which opens an input stream on the correct part in the multipart MIME message which curl constructed.

This is documented here:

http://docs.oracle.com/javaee/6/tutorial/doc/gmhba.html

Option B is probably the better option because it will work with forms and form uploads.