[英]BigQuery: Load Data into EU Dataset from GCS
In the past I have successfully loaded data into US-hosted BigQuery datasets from CSV data in US-hosted GCS buckets. 过去,我已成功将数据从美国托管的GCS存储桶中的CSV数据加载到美国托管的BigQuery数据集中。 We since decided to move our BigQuery data to the EU and I created a new dataset with this region selected on it.
此后,我们决定将BigQuery数据移至EU,然后我创建了一个新的数据集,并在其中选择了该区域。 I have successfully populated those of our tables small enough to be uploaded from my machine at home.
我已经成功填充了那些表,这些表足够小,可以从家里的机器上载。 But two tables are far too large for this so I would like to load them from files in GCS.
但是两个表太大了,因此我想从GCS中的文件中加载它们。 I have tried doing this from both a US-hosted GCS bucket and an EU-hosted GCS bucket (thinking that
bq load
might not like to cross regions) but the load fails every time. 我曾尝试从美国托管的GCS存储桶和欧盟托管的GCS存储桶中执行此操作(认为
bq load
可能不希望跨区域),但是负载每次都会失败。 Below is the error detail I'm getting from the bq
command line (500, Internal Error). 以下是我从
bq
命令行获取的错误详细信息(500,内部错误)。 Does anyone know a reason why this might be happening? 有谁知道发生这种情况的原因?
{
"configuration": {
"load": {
"destinationTable": {
"datasetId": "######",
"projectId": "######",
"tableId": "test"
},
"schema": {
"fields": [
{
"name": "test_col",
"type": "INTEGER"
}
]
},
"sourceFormat": "CSV",
"sourceUris": [
"gs://######/test.csv"
]
}
},
"etag": "######",
"id": "######",
"jobReference": {
"jobId": "######",
"projectId": "######"
},
"kind": "bigquery#job",
"selfLink": "https://www.googleapis.com/bigquery/v2/projects/######",
"statistics": {
"creationTime": "1445336673213",
"endTime": "1445336674738",
"startTime": "1445336674738"
},
"status": {
"errorResult": {
"message": "An internal error occurred and the request could not be completed.",
"reason": "internalError"
},
"errors": [
{
"message": "An internal error occurred and the request could not be completed.",
"reason": "internalError"
}
],
"state": "DONE"
},
"user_email": "######"
}
After searching through other related questions on StackOverflow I eventually realised that I had set my GCS bucket region to EUROPE-WEST-1
and not the multi-region EU
location. 通过在计算器上其他相关的问题,搜索后,我终于意识到,我已经把我的GCS斗区
EUROPE-WEST-1
而不是多区域EU
位置。 Things are now working as expected. 现在一切都按预期进行。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.