Redis质量增量更新

Question

I have about 10,000,000 records inside a redis database. I received a single columned CSV file with about 100,000 strings which correspond to keys in my redis database. For each of these strings inside the CSV I need to increment the value in redis by one. Normally to increment the INCR command is used, but is there a way I can make this faster than creating a loop that iterates 100,000 times and sends an INCR command one by one to change each key value individually? Is there a more mass way to update?

Answer 1

First of all, each redis driver has "Pipeline" to execute batch commands. You dont need to send the incr command one by one but send them together to redis server.

Second, if there are duplicate keys in your 100,000 strings, use "INCRBY" command. For instance, the doc is "k1,1; k2,2; k1,3", then you can use "INCRBY k1 2" instead of 2 "INCR k1"

Answer 2

Note: the following is pure speculation and needs testing to be verified :)

@Mark_H's answer is textbook (+1) but I have a wild idea that you can test if you want. Assuming (and that's a big assumption) that your 10M or so keys are serializable and that given the position of a key in the sequence you can derive the relevant key's name (eg if the names are based on a continuous numerical identifier) how about preparing a bitstring and have the set bits indicate an increment operation?

Such a bit value would be about 1.2MB in size, but the alternative is sending 100K ops (pipelined or not) so this is more network efficient. How about performance? The next part of the idea is to write a small Lua that actually accepts this value and does the INCR for relevant keys. I suspect it would perform equally as, if not better.

Keep us updated if you try this ;)

PS another hidden assumption in my answer is that you're only adding 1's, but that can be resolved by repeating the approach for other numbers/additional keys.