Needed clarification on how number of queues are defined in Netflix conductor

Question

I am a beginner to Netflix Conductor workflow and needed clarification on below.

If I have 3 instances of same workflow and all are ran together as follows.

Workflow Instance 1 - https://i.stack.imgur.com/tq0id.png

Workflow Instance 2 - https://i.stack.imgur.com/tq0id.png

Workflow Instance 3 - https://i.stack.imgur.com/tq0id.png

Since its a worker task, all tasks will be pushed into the queue: [W3-T1, W2-T1, W1-T1] (W - Workflow instance, T - Task no)

Now, when I poll once for the first task ie verify if idents are added test , I get the task of 1st workflow instance. I do post call to update the result of same.

Now, following is the current state of workflow instances:

Workflow Instance 1 - [2]: https://i.stack.imgur.com/Ip3FF.png

Workflow Instance 2 - https://i.stack.imgur.com/tq0id.png

Workflow Instance 3 - https://i.stack.imgur.com/tq0id.png

Here, the 2nd task of 1st workflow instance is also pushed to the queue. Our queue looks something like this: [W1-T2, W3-T1, W2-T1]

At this point, If I try to poll for Task2, I should be not getting W1-T2 as its not the first element of our queue. But I tried this and I got this task ie 2nd task of first workflow instance.

This means we don't have single queue or the organisation of queue is something different which I am not able to understand.

Can anyone help me understand how the no of queues are defined in conductor workflows.

Thanks in advance !!

Answer 1

Hi and thanks for the question!

You kind of hit on the answer in your question - there is not one single queue.

• Each Task has a Queue - so there is a queue for work to be done on T1, T2 and T3. These task queues are not strictly First in First out (FIFO) - as W1:T1 might take longer than W2:T1, so we'd see W2:T1 complete before W1:T1.

• Each workflow instance has a queue that ensures the order of the tasks Ex: (W1: T1 then T2 then T3). These are used internally by Conductor.