考虑延迟和效率来分配任务的最佳方式

Question

I'm looking for an algorithm to distribute some tasks. The problem is as follows:

Say I have a central task producer and some client consumers. The producer generates tasks and consumers take tasks (for starters, one at a time), process them, and when they are done, take new tasks (I already have a task queue).

The thing is, if you consider latency for a task to get from the producer to the consumer, it might make sense to group tasks together. For example, say we have 10 tasks in total and 2 consumers. If each of the tasks take 5 ms to get processed and the network latency is also 5 ms, sending 2 groups of 5 tasks each to each consumer will take 5ms + 5*5ms = 30ms, while sending the tasks individually would take 5*5ms + 5*5ms = 50ms, because the latency overhead appears for every task.

It's not as simple as grouping since some tasks will probably take longer, and it would make sense to send them separate as to let the other tasks that take a shorter time get processed in parallel by the other consumers. I'm planning on doing some statistics regarding the type of tasks. The number of consumers is also not constant.

Any idea of a good algorithm or a good read that can help me in achieving this?

Answer 1

At the moment a producer generates a task, not sending it right away will only increase that task's latency. Therefore I will assume that the task dispatcher works on snapshots of the current task queue: it takes all the tasks in the queue, sends them immediately in all directions, goes back to the queue, again takes all the tasks accumulated in the meantime, lather, rinse, repeat.

The dispatcher maintains an estimate of the completion time of each consumer. It orders the consumers according to increasing completion time and adds a task to the batch of the consumer with earliest completion time. Then it adds the average task time to that consumers completion time estimate, thus obtaining new estimate, then reorders the consumers according to the new estimates (in O(log n) using a heap) and goes to the next task. After all the tasks of the current snapshot are processed, send batches to the consumers and go make a new snapshot.

This policy will achieve equal consumer load on average . It can be improved:

• if each consumer is able to provide some feedback about the estimated completion time: it's the average task time multiplied by the number of tasks pending in the consumer. It's more precise because the consumer will use the actual time of the completed tasks instead of the average

• if the time to process each task is either known or can be estimated per-task, so the dispatcher will use a per-task estimate instead of an average.

EDIT: Forgot to mention:

The completion time is estimated as start-time + average-task-time * number-of-tasks-sent-to-a-consumer + latency * number-of-batches-sent-to-a-consumer .

Answer 2

To clarify my comment to your question, let's suppose you have the following loop in your consumer:

while (keepConsuming) {
    Task t = Task::get();
    t.process();
}

you could rewrite it like this (supposing we can use OpenMP):

Task cur=NULL, next;
do {
    #pragma omp sections
    {
        #pragma omp section
        if (cur != NULL) cur.process();
        #pragma omp section
        next = keepConsuming ? Task::get() : NULL;
    }
    cur = next;
} while (cur != NULL);

This way, the process() and get() inside the while are executed in parallel (obviously, assuming those two functions don't share any state).

Answer 3

Ahhh... the classic decision between fine-grained parallelism (which gives better load balancing but relatively higher overhead for synchronizing) versus coarse-grained parallelism (which obviously gives the opposite). Sorry but there's no easy answer...

Some thoughts:

1. Do lots of profiling, that's a good way to find a suitable number of tasks to group together. Just good old trial and error :)

2. Consider making a local task queue at each client. This can enable some sort of pre-fetch, eg when task n finishes, request task n+5 and start task n+1. Not sure if you are using multithreading or if task n+1 will be interrupted to accept task n+5.

3. Try to compact the task representation as much as possible. This may mean using char instead of int (this does make a difference for arrays). Maybe some parts of the task can be recalculated when it gets to the consumer.

4. Consider using some sort of timer on each consumer as feedback to adjust the number of tasks to take as a group next time. If you spend too much time, then grab fewer tasks next time. Beware a fancy heuristic may have some non-trivial overhead to it.

Answer 4

It seems that the main issue with the simple approach is that the consumer will stall for the amount of time it takes to fetch the next task. No useful work gets done during the stall.

Since latency -- rather than bandwidth -- is the main problem, one solution is to amortize the stall across multiple tasks, for example by grouping tasks into batches. To do this well, you need to have a good idea of how long each task will take to process.

An alternative is to fetch the next task in parallel with processing the current one. This can be easily done with two threads: thread A processing the current task and thread B fetching the next task. When A is done with the current task, the threads can either switch roles or pass the next task from B to A . This is a form of pipeline parallelism.

Answer 5

If your definition of latency can be augmented to 2 dimensions means that a cosumer can have different latency then you might try a space-filling-curve. A sfc subdivide the 2d and reduce the complexity to a 1 dimensiona. So you compute a number from f(x,y). Then you can sort this number and send the number in this order to the consumers. Of course you must write a SFC before you can use it I won't do it for you but I can help you if you have problems.