[英]From Big Query to Google cloud storage
I like to export data from Big query to Google cloud storage using any script.我喜欢使用任何脚本将数据从大查询导出到谷歌云存储。 Also for multiple table using loop save in CSV format and overwrite existing file.
同样对于使用循环保存的多个表,以 CSV 格式保存并覆盖现有文件。
Also how can we schedule this script.还有我们如何安排这个脚本。
If anybody have answer that will be great help.如果有人有答案,那将是很大的帮助。
Thanks in advance提前致谢
Common way to approach this problem is to use Airflow
and write a DAG to meet your requirements.解决此问题的常用方法是使用
Airflow
并编写一个 DAG 来满足您的要求。
But if you want to iterate tables and dump them in GCS on a regular basis only with BigQuery, following could be another option.但是,如果您只想使用 BigQuery 定期迭代表并将它们转储到 GCS 中,则以下可能是另一种选择。
1. Export Data 1.导出数据
You can export data to GCS with EXPORT DATA
statement in BigQuery script.您可以使用 BigQuery 脚本中的
EXPORT DATA
语句将数据导出到 GCS。
EXPORT DATA OPTIONS(
uri='gs://bucket/folder/*.csv',
format='CSV',
overwrite=true,
header=true,
field_delimiter=';') AS
SELECT field1, field2 FROM mydataset.table1 ORDER BY field1 LIMIT 10
2. Loops and Dynamic SQL 2.循环和动态 SQL
If you have a list of table you want to dump, you can loop those tables in BigQuery FOR
loop.如果您有要转储的表列表,则可以在 BigQuery
FOR
循环中循环这些表。
And you need to generate EXPORT DATA
script dynamically for each table.并且您需要为每个表动态生成
EXPORT DATA
脚本。 To do so, you can use EXECUTE IMMEDIATE
Dynamic SQL.为此,您可以使用
EXECUTE IMMEDIATE
Dynamic SQL。
3. Scheduling 3.调度
BigQuery provides a feature to schedule a user query and you can use it for your purpose. BigQuery 提供了一项功能来安排用户查询,您可以将其用于您的目的。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.