[英]Schedule a bq command line in GCP with Cloud Pub/Sub topic
I have a command line that I execute to select data from view and exporting to a staging table in BigQuery:我有一个命令行,我从视图中执行到 select 数据并导出到 BigQuery 中的暂存表:
bq mk --transfer_config --headless=true --project_id="XXXXXXX" --target_dataset=XXXXXX --display_name=XXXXXX --schedule='XXXXXX' --use_legacy_sql=false --params='{"query":"SELECT * FROM xxxxxxxx","destination_table_name_template":"xxxxxx","write_disposition":"WRITE_TRUNCATE"}' --data_source=scheduled_query --service_account_name="XXXXXXX@XXXXX.iam.gserviceaccount.com"
Is there any way to set the Cloud Pub/Sub topic using the command line?有什么方法可以使用命令行设置 Cloud Pub/Sub 主题吗?
Thank you in advance for the help!预先感谢您的帮助!
Obviously, by command line, you can't.显然,通过命令行,你不能。 It's possible by API call with this kind of payload
可以通过具有这种有效负载的 API 调用
{
"notificationPubsubTopic":"projects/PROJECT/topics/TOPIC",
"scheduleOptions": {
"disableAutoScheduling":false,
"startTime":"2020-01-24T10:06:12.344Z"
},
"displayName":"test",
"dataSourceId":"scheduled_query",
"destinationDatasetId":"DATASET",
"emailPreferences":{"enableFailureEmail":false},
"params": {
"query":"SELECT 1",
"write_disposition":"WRITE_APPEND",
"destination_table_name_template":"test_schedule"
},
"schedule":"every day 10:06"
}
This issue is a little tricky and not so obvious.这个问题有点棘手,不是那么明显。 I've found this discussion about it.
我找到了关于它的 讨论。
It seems that BigQuery's
CLI
or API
doesn't support this kind of notifications when creating the transfer configuration.在创建传输配置时,
BigQuery's
CLI
或API
似乎不支持这种通知。
As it was said in the discussion:正如讨论中所说:
This means we can't programmatically create transfers with notifications enabled :(
这意味着我们无法在启用通知的情况下以编程方式创建传输:(
You an find here the API's and CLI documentations regarding transfer service.您可以在此处找到有关传输服务的API和CLI文档。
I hope it helps you.我希望它能帮助你。
If you need any further information or support, don't hesitate about asking me.如果您需要任何进一步的信息或支持,请不要犹豫问我。
It seems that the Pub/Sub notification parameter has been added to the API and to the CLI (my CloudSDK version is 382.0).似乎已将 Pub/Sub 通知参数添加到 API 和 CLI(我的 CloudSDK 版本是 382.0)。 The parameter is
notification_pubsub_topic
.参数是
notification_pubsub_topic
。 In my case I need the email notification which is not added yet to the CLI (but to the API).在我的例子中,我需要 email 通知,它尚未添加到 CLI(但添加到 API)。 So I added it manually in
bigquery_client.py
in function CreateTransferConfig
.所以我在 function
CreateTransferConfig
中的bigquery_client.py
中手动添加了它。 For sure not the best way but it works for me.当然不是最好的方法,但它对我有用。
emailOptions = dict()
emailOptions['enable_failure_email'] = True
create_items['email_preferences'] = emailOptions
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.