ApplicationError: 10 on sending files to Google Cloud Storage using the Java remote API from App Engine

Question

I'm trying to migrate my old blobs from the Blobstore to the Could Storage, using the remote API to avoid nasty timeouts. But I'm stuck with the ApplicationError: 10 every time I close the GcsOutputChannel - with no uploaded files in the end.

My code:

//using the deprecated API to read the BlobStore:
FileService fileService = FileServiceFactory.getFileService();
AppEngineFile blobFile = fileService.getBlobFile(new BlobKey("myBlobKeyReadFromDatastore"));
FileReadChannel readChannel = fileService.openReadChannel(blobFile, false);
InputStream is = Channels.newInputStream(readChannel);

//now using the brand new API to write to the GCS:
GcsService gcsService = GcsServiceFactory.createGcsService();
GcsFilename filename = new GcsFilename("MyBucket", "theNameOfTheFileReadFromDatastore");
GcsFileOptions options = new GcsFileOptions.Builder().mimeType("theMimeTypeReadFromDatastore").build();
GcsOutputChannel writeChannel = gcsService.createOrReplace(filename, options);
OutputStream os = Channels.newOutputStream(writeChannel);

//and some reading -> writing...                
byte[] buff = new byte[1024 * 1024];
int read = 0;
while ((read = is.read(buff)) >= 0){
    os.write(buff, 0, read);
}
writeChannel.close(); //<-- Exception here!
readChannel.close();                    

The stacktrace:

com.google.appengine.tools.cloudstorage.RetriesExhaustedException: RetryHelper(13.39 s, 6 attempts, com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1@ddbef51): Too many failures, giving up
com.google.appengine.tools.cloudstorage.RetriesExhaustedException: RetryHelper(13.39 s, 6 attempts, com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1@ddbef51): Too many failures, giving up
    at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:115)
    at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:138)
    at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl.close(GcsOutputChannelImpl.java:140)
Caused by: java.io.IOException
    at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:620)
    at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:593)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:564)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:456)
    at com.google.appengine.api.files.FileWriteChannelImpl.close(FileWriteChannelImpl.java:81)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.append(LocalRawGcsService.java:159)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.finishObjectCreation(LocalRawGcsService.java:178)
    at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1.run(GcsOutputChannelImpl.java:143)
    at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1.run(GcsOutputChannelImpl.java:140)
    at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:93)
    ... 5 more
Caused by: com.google.apphosting.api.ApiProxy$ApplicationException: ApplicationError: 10: 
    at java.lang.Thread.getStackTrace(Thread.java:1567)
    at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:259)
    at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:68)
    at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:202)
    at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:199)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:199)
    at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:68)
    at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:107)
    at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:56)
    at com.google.apphosting.utils.remoteapi.RemoteApiServlet.executeRequest(RemoteApiServlet.java:384)
    at com.google.apphosting.utils.remoteapi.RemoteApiServlet.doPost(RemoteApiServlet.java:183)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:266)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
    at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:439)
    at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:435)
    at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:442)
    at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:186)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:306)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:298)
    at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:439)
    at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
    at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.apphosting.api.ApiProxy$ApplicationException: ApplicationError: 10: 
    at com.google.apphosting.utils.runtime.ApiProxyUtils.convertApiResponseRpcErrorToException(ApiProxyUtils.java:108)
    at com.google.apphosting.utils.runtime.ApiProxyUtils.convertApiError(ApiProxyUtils.java:70)
    at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:493)
    at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:411)
    at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:811)
    at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:442)
    at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:186)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:306)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:298)
    at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:439)
    at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.runCallback(RpcStub.java:849)
    at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:859)
    at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:838)
    at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:984)
    at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:889)
    at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1291)
    at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:1124)
    at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2690)
    at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2433)
    at com.google.net.rpc3.impl.client.RpcNetChannel.access$2300(RpcNetChannel.java:159)
    at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3605)
    at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:640)
    at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:444)
    at com.google.apphosting.runtime.udrpc.UdrpcTransport$ClientAdapter.receivedMessage(UdrpcTransport.java:567)
    at com.google.apphosting.runtime.udrpc.UdrpcTransport.dispatchPacket(UdrpcTransport.java:386)
    at com.google.apphosting.runtime.udrpc.UdrpcTransport.access$500(UdrpcTransport.java:63)
    at com.google.apphosting.runtime.udrpc.UdrpcTransport$5.run(UdrpcTransport.java:286)
    at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260)
    at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121)
    at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:579)
    at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:994)
    at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:876)
    at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:160)
    at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1849)

The error occurs even with files with less than 1KB in size. My Google Cloud Storage is configured with the user <my-app-id>@appspot.gserviceaccount.com as owner of the bucket and as "can edit" in the project. The Billing is enabled as well (both in App Engine and in the Google Cloud Console). I'm using the App Engine SDK 1.8.3.

I don't know what to do. Any ideas?

Answer 1

This was a bug. It was fixed in this change: https://code.google.com/p/appengine-gcs-client/source/detail?r=104 Sync to the latest version in the SVN and it should go away.