Multi-threading a stage in the pipeline pattern with ExecutorService

Question

I have a multi-stage pipeline. Each stage runs in a separate thread with communication happening using bounded BlockingArrayQueues. I'm trying to multi-thread the slowest stage to improve throughput.

Question: What is the recommended implementation for this? Is there a library that would make this implementation simpler & easier to read?

Input Queue -> Stage 1 (4 threads) -> Bounded Queue -> Stage 2

Requirements :

• Work units on the input queue are independent.

• Work units are strictly ordered -- output should be in the same order as input.

• Stage 1 must be throttled -- it must stop if the output exceeds a certain size.

• Exceptions in Stage 1 should result in a "poison pill" on the output queue and terminate the ExecutorService. Queued tasks should be discarded best effort.

** My proposed implementation:**

I'm thinking of using a ThreadPoolExecutor with a limited # of threads.

Strict ordering will be enforced with a CountDown latch on each work unit. A thread can only push the result if the previous work unit's latch is 0 and there's space on the queue. This also takes care of throttling since the thread will block until there's room on the output queue.

class WorkUnit {
   CountDownLatch previousLatch;
   CountDownLatch myLatch;
}

class MyRunnable extends Runnable {
   public void run() {
       //do work...
       previousLatch.await();
       ouputQueue.put( result );
       myLatch.countDown();
   }
}

Exception handling is where I'm a little stumped. I'm thinking of overriding ThreadPoolExecutor.afterExecute() which will call shutdownNow() if there's an exception.

class MyThreadPoolExecutor extends ThreadPoolExecutor {
      protected void afterExecute(Runnable r, Throwable t) {
           if(t != null) {
               //record exection, log, alert, etc
               ouput.put(POISON_PILL);
               shutdownNow();
           }
      }
}

Answer 1

Using ExecutorService requires totally asynchronous design, without using blocking operations like CountDownLatch.countDown() or BlockingQueue.take() . When an action B must wait some event(s), then that event(s) should start action B by submiting a Runnable to the ExecutorService .

In you case, you should create custom classes instead of the queues. That classes should accept messages and either store them internally, or submit tasks which implement Stage1 or Stage2, according to some rules (eg limit the number of running Stage1 tasks).

As for ordering, replace reference to previous task with serial number.