Data fanout Java library

Question

This question relates to a very common problem that I haven't been able to find a conventional solution for.

Here is the setup:

1. You have a number of consumers, each subscribing to a set of symbols
2. You have a number of producers each producing data for a disjoint subset of these symbols
3. Consumers may be too slow to consume all changes to the symbols they subscribed for so you may need to throttle
4. Consumers are only interested in the most recent datum for each symbol. If a consumer missed an update for a symbol, and a newer datum is available then only the newest one should be sent.

I've run into this problem quite frequently and each time had to reinvent the wheel, for instance implementing a queue in which unconsumed data can be replaced by newer data. I'm wondering if there are some libraries which implement a solution to this in an efficient manner.

Answer 1

Sounds like your publishing out market data feeds and you want clients to subscribe to specific feeds, aside from that point, you don't need a queue as you don't need to process every data message. Use UDP as your transport protocol to publish out the market data as UDP does not not require its packets to be confirmed as being received before it sends out the next packet. Clients should just cache the last value they receive and there is no need to maintain a queue. You can then have an observer on this last value and publish it out to the rest of your applications when it changes.