How to call a function from another file in Python for PythonOperator

Question

I'm writing a Airflow DAG that uses a PythonOperator. It requires a "python_callable" which it seems to be a function. I'm calling this Python code dag-transf.py

I have a Python code that does some transformation through pandas (transf.py)

I need dag-transf.py task to execute transf-py as the "python_callable" parameter.

with DAG(dag_id='dag_transformation',
         default_args=default_args,
         catchup=False,
         schedule_interval="0 0 1 * *") as dag:

    exporting = PythonOperator(
        task_id='exporting_task',
        python_callable=transf_function #transf_function needs to actually be transf.py

How am I supposed to use this external file as the function code itself?

Answer 1

You can read the https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html docs.

You really need to define a package/module where your function can be imported from. This is standard Python way of importing and using functions. The only 'airflow` related thing is where you should create packages and modules - they all should be creaed in the "dags folder where you can import them from (but ready the doc above to get details of it.

Answer 2

Create a python file in the DAG Directory.

• dag_trans.py

Write your transf_function inside the created file.

In your dag, import your python package

from dag_trans import transf_function

Then use:

python_callable=transf_function