将文件上传到Google Cloud Storage：500后端错误

Question

我们正在尝试将文件上传到Google Cloud Storage中，然后再将其移到BigQuery中，但是在某些上传过程中，我们经常会遇到“ 500 Internal Server Error”或“ 410 Gone”（以下是原始消息）。

我们使用的是官方SDK，并添加了带指数补偿的重试功能，但错误始终存在。 请问您有什么建议吗？

这是我们上传（scala）的方式：

val credential = new GoogleCredential().setAccessToken(accessToken)

val requestInitializer = new HttpRequestInitializer() {
    def initialize(request: HttpRequest): Unit = {
        credential.initialize(request)

        // to avoid read timed out exception
        request.setConnectTimeout(200000)
        request.setReadTimeout(200000)

        request.setIOExceptionHandler(new 
            HttpBackOffIOExceptionHandler(new ExponentialBackOff()))
        request.setUnsuccessfulResponseHandler(new 
            HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackOff()))
    }
}

val storage = new Storage.Builder(
    new NetHttpTransport,
    JacksonFactory.getDefaultInstance,
    requestInitializer
).setApplicationName("MyAppHere").build

val objectMetadata = new StorageObject()
    .setBucket(bucketName)
    .setName(distantFileName)

val isc = new InputStreamContent("binary/octet-stream", fis)
val length = isc.getLength
val insertObject = storage.objects().insert(bucketName, objectMetadata, isc)
// For small files, you may wish to call setDirectUploadEnabled(true), to
// reduce the number of HTTP requests made to the server.
if (length > 0 && length <= 2 * 1000 * 1000 /* 2MB */ ) {
    insertObject.getMediaHttpUploader.setDirectUploadEnabled(true)
}
insertObject.execute()

我们的scala依赖关系：

"com.google.api-client" % "google-api-client" % "1.18.0-rc",
"com.google.api-client" % "google-api-client-jackson2" % "1.18.0-rc",
"com.google.apis" % "google-api-services-bigquery" % "v2-rev142-1.18.0-rc",
"com.google.apis" % "google-api-services-storage" % "v1-rev1-1.18.0-rc",
"com.google.http-client" % "google-http-client" % "1.18.0-rc",
"com.google.oauth-client" % "google-oauth-client" % "1.18.0-rc"

原始SDK错误响应：

500 Internal Server Error
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

410 Gone
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

Answer 1

每个Backend error都应进行指数重试，因为可能存在服务问题。

如果在10个小时后错误仍然存​​在，那么您应该与支持人员联系，以便为您的问题提供1：1的帮助。