具有溢出到磁盘的Java FIFO队列

Question

I'm working/preparing an application that is based on a producer/consumer model. In my case there will be one producer which generates several million (non-trivial) tasks, and there will be a configurable number of consumers.

Communication between producer and consumers is basically Queue-based. I'm worried however about memory-consumption: it's quite conceivable that the number of tasks will exceed the available memory for the JVM. So I'd like to have a Queue implementation that only keeps the "top-X" number of queue items in memory, and stores the rest on disk. This does not have to be resilient in that it does not need to survive a restart of the program.

I've searched around, but can't find a Queue implementation that seems to be in widespread use (there do seem to be a few, what I call, proof-of-concept implementations, but I'm worried about future support/continued development of those implementations). I'm also looked at external Messaging Queue applications, but (1) I don't want to run a second external process and (2) even embedding a message broker inside the same JVM process seems a bit "top heavy" for this requirement.

Does anybody know of any well-supported future-proof library out there that provides this functionality?

Rgds

Answer 1

Well, JMS seems like the obvious solution. I don't think you'll find something solid to solve this problem since JMS solves it and is a standard solution.

Note however that Java also has BoundedQueues to solve this problem: dimension the queue to make sure it won't fail with an OOME when the queue is full, and producers will be blocked while trying to put a message in the full bounded queue until some task is removed from the queue by one of the consumers.

Answer 2

Having the producer block when there is more than enough tasks to be consumed is usually more efficient than letting the queue grow and consume more memory. eg if your queue fits in the queue it can be several times faster than a queue which doesn't.