On-disk lookup table with node.js bindings

Question

For a project I am creating a queuing library and basically store URLs in a Set (it's actually an object, where I set keys to true, but one can see it as an array), so the queue only takes every url once. This works really well, however I am facing the problem that there are many URLs and so the RAM usage becomes really high.

Therefor I want to use an on-disk key-value store (actually only keys are required, no idea whether there is some different approach) with the following requirements:

• No need to load the whole data set into RAM
• Speedy lookups
• Node.js bindings

It doesn't have to be too safe (losing data once in a while isn't a huge problem, low RAM requirements are more important) and even though I use Node.JS in this scenario this lookup doesn't necessarily need to run async.

Actually a side question would be whether there is some better way than a on-disk key-value approach. A term would be nice. Lookuptables somehow always lets me find data sets (IPs, ZIP codes, etc.)

Answer 1

I'd use a sql table with a single column (to store the url). Better control on memory usage than redis (which pretty much stores all in memory).

• easy to check if there is already the same value
• easy to insert
• easy to remove one element

Answer 2

If it really "doesn't have to be too safe", another design would be to keep storing everything in memory but limit the number of URLs you store, for example by using an LRU cache.

You could either use a cache in node.js (easy to find via Google ) or use a separate memcached server, possibly on the same machine.