将 airflow 连接到谷歌融合

Question

I'd like to write python script which manages my google data fusion pipelines and instances (creates new, deletes, starts, etc). For that purpose I use airflow installed as library. I've read some tutorials and documentations but I still can't make that script connect with data fusion instance. I've tried to use next string:

export AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT='google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fkeys%2Fkey.json&extra__google_cloud_platform__scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform&extra__google_cloud_platform__project=airflow&extra__google_cloud_platform__num_retries=5'

with my data json key file and Project id but it still doesn't work. Can you give me an example of creating that connection?

Answer 1

You can find an example python script here: https://airflow.readthedocs.io/en/latest/_modules/airflow/providers/google/cloud/example_dags/example_datafusion.html

This page provides a breakdown for each Data Fusion Operator if you would like to learn more about them: https://airflow.readthedocs.io/en/latest/howto/operator/gcp/datafusion.html