[英]Unable to load a JSON file from google cloud storage to bigquery using python
I am attempting to export a file from my Google cloud storage to Google bigquery but am running into numerous problems. 我正在尝试将文件从我的Google云存储导出到Google bigquery,但遇到许多问题。
#standardSQL
import json
import argparse
import time
import uuid
from google.cloud import bigquery
from google.cloud import storage
dataset = 'dataworks-356fa'
source = 'gs://dataworks-356fa-backups/pullnupload.json'
def load_data_from_gcs(dataset, test10, source ):
bigquery_client = bigquery.Client(dataset)
dataset = bigquery_client.dataset('FirebaseArchive')
table = dataset.table(test10)
job_name = str(uuid.uuid4())
job= bigquery_client.load_table_from_storage(
job_name, table, "gs://dataworks-356fa-backups/pullnupload.json")
job.source_format = 'NEWLINE_DELIMITED_JSON'
job.begin()
# wait_for_job(job)
print("state of job is: " + job.state)
# print("errors: " + job.errors)
load_data_from_gcs(dataset, 'test10', source)
When the wait_for_job(job)
line is not commented out I receive this error 当wait_for_job(job)
行未注释掉时,我收到此错误
Traceback (most recent call last):
File "cloudtobq.py", line 42, in <module>
load_data_from_gcs(dataset, 'test10', source)
File "cloudtobq.py", line 38, in load_data_from_gcs
wait_for_job(job)
NameError: global name 'wait_for_job' is not defined
And when print("errors: " + job.errors)
is not commented out I receive this error. 当未注释掉print("errors: " + job.errors)
时,我会收到此错误。
Traceback (most recent call last):
File "cloudtobq.py", line 42, in <module>
load_data_from_gcs(dataset, 'test10', source)
File "cloudtobq.py", line 40, in load_data_from_gcs
print("errors: " + job.errors)
TypeError: cannot concatenate 'str' and 'NoneType' objects
When both are commented out this is what I receive and then being returned to the original terminal screen. 当两者都被注释掉时,这就是我收到的内容,然后返回到原始终端屏幕。
Wess-MacBook-Pro:desktop wesstephens$ python cloudtobq.py
state of job is: RUNNING
Wess-MacBook-Pro:desktop wesstephens$
You need to include the definition of the function from the documentation sample code : 您需要从文档样本代码中包括函数的定义:
def wait_for_job(job):
while True:
job.reload()
if job.state == 'DONE':
if job.error_result:
raise RuntimeError(job.errors)
return
time.sleep(1)
You don't need to print job.errors
, since wait_for_job
will raise an exception if the job was unsuccessful. 您不需要打印job.errors
,因为如果作业失败,则wait_for_job
将引发异常。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.