如何将 BigQuery 视图作为 csv 文件传输到 Google Cloud Storage 存储桶

Question

I need to export the content of a BigQuery view to the csv file in GCP, with Airflow DAG. To export the content of the BQ TABLE, I can use BigQueryToCloudStorageOperator. But in my case I need to use an existing view, and BigQueryToCloudStorageOperator fails with this error, which I see while checking the logs for failed DAG:

BigQuery job failed: my_view is not allowed for this operation because it is currently a VIEW

So, what options do I have here? I can't use a regular table, so may be there is another operator that would work with a view data stored in BQ, instead of table? Or may be the same operator would work with some addition options (although I don't see anything useful in here Apache documentation for BigQueryToCloudStorageOperator )?

Answer 1

I think the Bigquery client doesn't give the possibility to export a view to a GCS file.

It's not perfect but I propose you 2 solutions

First solution (more native with existing operators):

• Create a staging table to export it to GCS
• At the beginning of your DAG, create a task that truncate this staging table
• Add a task with a select on your view and an insert in your staging table ( insert/select )
• Use the bigquery_to_gcs operator from your staging table

Second solution (less native with Python clients and PythonOperator):

• Use a PythonOperator
• In this operator, use a Bigquery Python client to load data from your view as Dict and the storage Python client to generate a file to GCS from this Dict

I have a preference for the first solution, even if it forces me to create a staging table.

Answer 2

I ended up with a kind of combined solution, part of it is what Mazlum Tosun suggested in his answer: in my DAG I added an extra first step, a DataLakeKubernetesPodOperator , which runs a Python file. In that Python file there are calls to SQL files, which contain simple queries (put in the await asyncio.wait(...) block and executed with bq_execute() ): truncate an existing table (to prepare it for a new data), and then copy (insert) data from the view to the truncated table (as Mazlum Tosun suggested).

After that step, the rest is the same as before: I use BigQueryToCloudStorageOperator to copy data from the regular table (which now contains data from the view) to google cloud storage bucket, and now it works fine.