根据安慰队列中的安慰事件触发气流任务

Question

I have just started learning airflow...!!

I need to create DAG which contains task to listen to events from solace queue, There could be multiple events that task is going to listen based on event(message) it should pass to next task.

Once Event listener listen event E1 then it should trigger "Event E1" task and it would start listening to next event.

Same goes for other Events(tasks), once listener receive all events then it should turn green(succeed).

I know on the basis of event we can trigger next job by branching, but how we can make sure first task is running until all events are received?

Answer 1

I haven't tested it, but this should give you the structure you're looking for I believe.

# wait_eN = Your event waiting tasks
# work_eN = Your event working tasks

wait_e1 >> (wait_e2, work_e1)
wait_e2 >> (wait_e3, work_e2)
wait_e3 >> work_e3

Answer 2

We have developed solace event listener which listen event from solace queue and create event Json file. From Airflow we have implemented sensor operator with reschedule on. With this approach we are able to meet our requirements.