生产者消费者模式-处理生产者失败

Question

I have a producer/consumer pattern like the following

• A fixed number of producer threads, each writing on to their own BlockingQueue, invoked via an Executor
• A single consumer thread, reading off the producer threads

Each producer is running a database query and writing the results to its Queue. The consumer polls all the producer Queues. At the moment if there is a database error the producer thread dies and then the consumer gets stuck forever waiting for more results on the product Queue.

How should I be structuring this to handle catch errors correctly?

Answer 1

I once did a similar thing and decided to use a sentinel value that the dying producer thread would push into the queue from the catch-block. You can push the exception itself (this works in most scenarios), or have a special object for that. In any case it is great to push the exception to the consumer for debugging purposes.

Answer 2

It appears that the only option you have when a producer dies is to stop the consumer.

To do this you can use a poison pill. This is a special object which the producer adds when it stops and the consumer knows to stop when it receives it. The poison pill can be added into a finally block so it is always added no matter how the producer is killed/dies.

Given you have only one consumer, I would use one queue. This way your consumer will only block where all the producers have died.

Answer 3

Whatever class it is that you actually push onto the queue/s, it should contain success/fail/error members so that the consumer/s can check for fails.

Peter has already suggested using only one queue - I don't see how avoiding all that polling should be any particular problem - the objects on the queue can have members that identify which producer they came from, and any other metadata, if required.

Answer 4

You might add some timeout to kill the consumer when there are no more elements in the queue(s) for a certain time.

Another approach might be to have the producers maintain an "alive" flag and signal that they are dying by setting it to false. If the producers run continuously but might not always get results from the database the "alive" flag could be the time the producer reported to be alive the last time and then use the timeout to check whether the producer might have died (when the last report of being alive was too long ago).

Answer 5

Answering my own question.

I used the following class. It takes a list of Runnable and executes them all in parallel, if one fails, it interrupts all the others. Then I have interrupt handling in my producers and consumers to die gracefully when interrupted.

This works nicely for my case.

Thanks for all the comments/answers as they gave me some ideas.

// helper class that does the following
//
// if any thread has an exception then interrupt all the others with an eye to cancelling them
// if the thread calling execute() is interrupted then interrupt all the child threads
public class LinkedExecutor
{
    private final Collection<Runnable> runnables;
    private final String name;

    public LinkedExecutor( String name, Collection<Runnable> runnables )
    {
        this.runnables = runnables;
        this.name = name;
    }

    public void execute()
    {
        ExecutorService executorService = Executors.newCachedThreadPool( ConfigurableThreadFactory.newWithPrefix( name ) );

        // use a completion service to poll the results
        CompletionService<Object> completionService = new ExecutorCompletionService<Object>( executorService );

        for ( Runnable runnable : runnables )
        {
            completionService.submit( runnable, null );
        }

        try
        {
            for ( int i = 0; i < runnables.size(); i++ )
            {
                Future<?> future = completionService.take();
                future.get();
            }
        }
        catch ( InterruptedException e )
        {
            // on an interruption of this thread interrupt all sub-threads in the executor
            executorService.shutdownNow();

            throw new RuntimeException( "Executor '" + name + "' interrupted", e );
        }
        catch ( ExecutionException e )
        {
            // on a failure of any of the sub-threads interrupt all the threads
            executorService.shutdownNow();

            throw new RuntimeException( "Execution execution in executor '" + name + "'", e );
        }
    }
}