GAE Upload large file to blobstore from an URL

Question

I want to upload large video file 30MB to 2GB to the blob store having an external URL to access to this file.

So far i tried something like this but it doesn't work for large file, I think because of the 32MB limit:

    //InputStream is = new URL(getUrlThumbnailFullSize(wistiaResponse)).openStream();
    HttpURLConnection con = (HttpURLConnection) new URL("URL of the file here").openConnection();
    con.setConnectTimeout(0);
    con.connect();
    InputStream is = con.getInputStream();
    //create file service to upload the video to the blob store
    FileService fileService = FileServiceFactory.getFileService();
    //create the App Engine File; assign MIME type and name
    AppEngineFile file = fileService.createNewBlobFile("content type here", "file name here");
    //lock when uploading file
    boolean lock = true;
    //open channel
    FileWriteChannel writeChannel = fileService.openWriteChannel(file, lock);
    //upload file
    writeChannel.write(ByteBuffer.wrap(IOUtils.toByteArray(is)));
    writeChannel.closeFinally();
    //release the lock
    lock = false;
    //get blob key
    BlobKey blobKey = FileServiceFactory.getFileService().getBlobKey(file);

But i have the following stack trace:

SEVERE: com.google.appengine.api.urlfetch.ResponseTooLargeException: The response from url http://embed.wistia.com/deliveries/a0f8dc8ed210b53152af601eb0af40d76cd2929f.bin was too large. at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:57) at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417) at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296) at controller.CtrlWistia.download(CtrlWistia.java:149) at controller.CtrlWistia.ajax(CtrlWistia.java:89) at controller.CtrlAjax.main(CtrlAjax.java:66) at vidaao.AjaxServlet.doPost(AjaxServlet.java:37) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.d oRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:61) at org.mortbay.jetty.servlet.Servlet Handler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.BackendServersFilter.doFilter(BackendServersFilter.java:97) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContex t.java:418) at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:94) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:409) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326)

Answer 1

Please use blobstoreService.createUploadUrl for large file upload.

[Updated] Ah, I think I got it. Currently it's not possible since URLFetch service has 32MB response size restriction. As a workaround, can you download the file from the URL within the client side javascript and upload it with the upload URL created by blobstore service?

[Updated] Okay, you need to achieve this on server side, which was not clear to me from the question. So please forget about createUploadUrl.

If the server supports Range header, you can request smaller chunk of data multiple times and store them as separate blob store objects, alongside with the required metadata which would be filename, mimetype, list of blob keys, etc.

Additionally, if you want to combine them into a single file, you may use Google Cloud Storage. With Resumable upload option , you can send chunk of data multiple times to Google Cloud Storage.