NoSQL建议项目（文本流）

Question

I'm looking for a NoSQL DB recommendation... here's what I'm working on:

I'm writing a web-based client for delivering text streams (basically, real-time captions) to a significant number of consumers. Once things are fully ramped up, there might be 100+ events happening at any given moment. Many will be small (< 10 consumers) but some of them could be quite large (10,000+ simultaneous consumers, maybe more?).

During the course of each event, text will be accumulating at a rate of anywhere from a few words per minute up to 200+ words per minute. Each consumer will be running a web client (a browser on a desktop/laptop/tablet/smartphone) which will poll periodically for any text that it hasn't already received. It will also be possible for a given user to ask for the full text of the event up to the time that they make the request. Completed events have to stick around for a while, but will be removed within about 24-36 hours of their completion.

My first thought is to use Redis, which has methods for appending to a text value in the datastore as well as built-in support for getting a substring from the end of a text value (ie a client could just hold the character offset of the last character it received and would pass that to the client API and that would be used to pull a substring from the event text). I am concerned though that the growth of the string containing the event text might be an unusual use of Redis and could cause me some issues.

So... is there a NoSQL DB that seems particularly well suited to this sort of application? Is there any significant reason NOT to use Redis for something like this?

Answer 1

An underlying open question is what to do about new clients. For example, say an event has started and someone connects a few minutes into it. Do they need everything from the beginning or just from when they connected?

If the latter I'd recommend a message system instead of appending strings to strings. One way would be to use Redis' Pub/Sub instead. That seems a better fit overall, and especially if new connections do not need everything from the beginning. For longer term storage, a client that listens as any other and the archive entry - preferably by local cache and then upload the completed transcript when completed or in-progress. I'd keep the real-time need and code separate from requesting history and archives.

Another route would be to use an ordered set, using a timestamp for the time the entry was made. As a result the client only keeps track of the last update and retrieves anything from that time on. Ordered Sets documentation can be found here . This method also provides the ability to select a region of time from the transcript. With a bit of math you could even replay the event from a transcript viewpoint as if it were live. If you've got tens of thousands of clients pulling the entire transcript each poll

Another advantage of the timestamp ordered set is string encoding. When using Redis strings and getrange you have to use fixed-width encodings. The range is byte-offsets, not character offsets. If you need the ability to support, say UTF-8, this might be a problem for you.

A third option is to append a string of text to a list. This is similar to the sorted set except that your client stores the last index (size of the list) and on each poll tries to get anything from lastIndex+1 to the end.