Passing a command line argument (argument parser) to airflow PythonOperator

Question

Is there a way to pass a command line argument to Airflow PythonOperator. Currently, I have a python script that accepts a few arguments and performs some specific activities. I am calling the method into the dags from the script. I initially planned to use op_args in the dags, but it seems to not work as I get following error

{standard_task_runner.py:107} ERROR - Failed to execute job 56 
for task test (test() takes 1 positional argument but 4 were given; 544)

If I were to give the arguments in command line, it would be some like python test.py -f Hello -b World and it would print me Hello World . I give the script I have written for DAG is below:

from airflow.operators import PythonOperator
from airflow.models import DAG
from datetime import datetime, timedelta
from test import test
default_args = {
    'owner': 'none',
    'retries': 5,
    'retry_delay': timedelta(minutes=5)
}
with DAG(
    dag_id='test',
    default_args = default_args,
    description =' Still in process',
    start_date=datetime(2022,9,13),
    schedule_interval='@weekly'

) as DAG:
    task = PythonOperator(
        task_id='test',
        python_callable=test,
        op_args = ['-f','Hello','-b','World']
    )

task

I tried looking into Airflow Docs , but it doesn't seem to have any examples to provide op_args (they only have op_kwargs ). Not only that, but I also looked into some other stack overflow questions like this one, but it uses bash operator, but I want to use python operator

Edit 1: as someone asked for the test function, i write it below

import argparse
import sys
def test(args):
    
    parser = argparse.ArgumentParser(description='Description of your program')
    parser.add_argument('-f','--foo', help='Description for foo argument', required=True)
    parser.add_argument('-b','--bar', help='Description for bar argument', required=True)
    args = parser.parse_args(args)
    if args.foo == 'Hello':
        # code here
        print('hello')

    if args.bar == 'World':
        # code here
        print('world')

if __name__ =='__main__':
    test(sys.argv[1:])

I can run the script above by running python.\airflow\dags\test.py -f Hello -b World in cmd prompt OR import this file and call the function test(['-f','Hello','-b','World']) in another function

Answer 1

Your test function is getting one parameter (args) and not list (*args). in that case, try to pass op_args = [['-f','Hello','-b','World']]