Airflow如何从python运算符创建数据流作业？

Question

When I am running my Beam pipeline by the command line, using the direct runner or the dataflow runner, it works fine...

Example:

$ python my_pipeline.py --key /path/to/gcp/service/key.json --project gcp_project_name

But when I am trying to use airflow, I have two options, bash operator or python operator.

Using bash operator will success, but will limit my ability to use the airflow features.

But what I am trying to do is to run it as a python operator. So I am importing the module inside the airflow dg file, and then run it as a python operator.

It is also working fine if I am using the local runner, but when I changed it to the data flow runner, it fails after creating the job on GCP dataflow with this error

ImportError: No module named airflow.bin.cli

What I am missing to let Airflow create a dataflow job from a python operator?

Answer 1

OK, That is not the perfect solution to it, but you can use

DataFlowPythonOperator()

which will run the exact same bash command we mentioned before. It is a workaround and not equal to the PythonOperator but more like running a BashOperator ... Still can't use the strength of Airflow features in the current case (like xcom)... Docs