How to run the same Python script multiple times using Airflow?

Question

I'm trying to run my Differential Evolution script in Python called differential_evolution.py . Each iteration runs for around 40 generations. I want to run 50 iterations in parallel using Airflow. I have provided random seed in my script so that each iteration creates different results.

Snippet of differential_evolution.py :

Optimizer() is a custom class I created to run the algorithm. solution stores the solution list in list attribute x . And mape calculates the mape for the solution list x .

for iteration in range(50):
    seed = np.random.randint(0, 1000)  
    opt_obj = Optimizer()  
    solution = opt_obj.run_optimizer()  
    mape = opt_obj.calc_performance(solution.x)

Each iteration creates two output files: abc.txt and xyz.csv to store relevant information for different variables.

Snippet of the dag script :

start >> create_cluster >> differential_evolution.py >> delete_cluster >> end

This is running fine but taking lots of time when you run for 50 iterations.

---

What I want is to create a dag like this:

start >> create_cluster >> [iteration 1, iteration 2, ... iteration 50] >> delete_cluster >> end , where each iteration outputs the same two files abc_ i .txt and xyz_ i .csv ( i is the ith iteration)

Answer 1

You could do something like this:

@task
def task1():
    something here

@task
def task2():
    something here

@task
def task_optimizer():
   seed = np.random.randint(0, 1000)  
   opt_obj = Optimizer()  
   solution = opt_obj.run_optimizer()  
   mape = opt_obj.calc_performance(solution.x)


start_task=task1()
end_task=task2()

for iteration in range(50):
    optimize_task = task_optimizer()
    start_task >> optimize_task >> end_task

    

You would need to adjust so the optimizer generates random numbers properly but something along this line should work.