Schedule loading data from GCS to BigQuery periodically

Question

I've researched it and currently come up with a strategy using Apache Airflow. I'm still not sure how to do this. The most blogs and answers I'm getting are directly codes instead of some material to better understand it. Also, please suggest if there is a good way to do it.

I also got an answer like using Background Cloud Function with a Cloud Storage trigger .

Answer 1

You can use BigQuery's Cloud Storage transfers , but note that it's still in BETA.

It gives you the option to schedule transfers from Cloud Storage to BigQuery with certain limitations.

Answer 2

The most blogs and answers I'm getting are directly codes

Apache Airflow comes with a rich UI for many tasks but that doesn't mean you are not supposed to write code in order to get your task done.

For your case, you need to use BigQuery command line operator for Apache Airflow

A good way on how to do this can be found in this link