DataJobs Airflow 用于 GCP 的 Dags 和 Azure

Question

Implement Operator to abstract cloud provider (GCP & Azure) -specific calls (SQL/Python/Spark)

I have found this documentaion:- https://airflow.apache.org/docs/apache-airflow/1.10.14/_modules/airflow/contrib/example_dags/example_gcp_sql.html

But i am not clear how to implement this according to my task

Answer 1

You might want to look through the available operators of the Google and Azure Airflow providers if an operator exists that fits your use case.

The Astronomer registry has some example DAGs with implementations of some of the operators in those packages (Google example DAGs , Azure example DAGs )

Disclaimer: I work at Astronomer:)

PS: The link you shared points to very outdated documentation of pre-2 Airflow. It is very possible for that DAG code to throw errors in modern Airflow.

EDIT:

• If this is your first time using Airflow I recommend you start with this tutorial . Part 1 teaches you about operators and how to interact with Airflow, including how to turn any Python script into an Airflow task. Part 2 shows you how to create connections with external tools, which you'll need when interacting with GCP and Azure.
• This guide goes over how to use several SQL based operators with full DAG examples.