optimizing querying multi keys in redis via node

Question

I have the certain key maintained in redis in the following format

identifier:username:userid:categoryname

Eg:

Blacklist:tomhanks:12345:movies
Blacklist:tomhanks:12345:music
Blacklist:micaheljordan:23456:sports
Blacklist:micaheljordan:23456:movies

Now there are times I have username available with me and in some case userid with me. But then I don't know which one would be available when. As per the API request I can have either of them. So i need to query it twice - once with username - secondly with user id.

Eg: KEYS Blacklist:tomhanks:* & KEYS Blacklist:*:12345:* and then combine the result and accordingly blacklist the category fetched from some other API.

I am using nodejs to query this redis instance, so rather than querying it twice I decided to use multi exec in node js

As per the example on this page node redis github example for multi command , i am querying the redis instance in the following format

client.multi([
    ["keys", "Blacklist:tomhanks:*"],
    ["keys", "Blacklist:*:12345:*"]
]).exec(function (err, replies) {
    console.log(replies.toString());
});

But then recently I checked by slowlog in redis and I found exec commands among the top of the list and also some of the queries timing out. Although the KEYS are just 10000 in number.

Also intermittently i am getting console.log messages node_redis: no callback to send error: ERR EXEC without MULTI

POA:

1. use sets rather than KEYS in the following format KEY: Blacklist:tomhanks -> movies, music KEY: Blacklist:123456 -> movies, music
2. or somehow restructure the 2nd key Blacklist:*:12345:* which has 2 wildcards and hence comparatively heavy Suggested structure would be Blacklist:tomhanks:movies Blacklist:tomhanks:music Blacklist:12345:movies Blacklist:12345:music

As per the warning posted on this page about KEYS

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using sets.

Is using SETS a better idea than KEYS ?

Answer 1

Never ever use the KEYS command is it completely blocks the entire database (Redis has a single worker thread for handling commands) from responding for duration of the command ( just like any other command). The difference with KEYS is that it scans and return the entire data-set at once. For large data-sets, it can prove to be problematic.

Redis SCAN commands are your friends. SCAN works by iterating over a matched result set ( supports glob matching such as KEYS). until all results have been returned. This iterative approach allows for 'fair' resource distribution when querying large data sets, as each iteration handles a smaller portion of the data and 'competes' with other commands for cpu time.

You don't have to use SETS in order to scan elements( though there is the SSCAN variant), but it will have implications for the future ( going cluster or staying in standalone/sentinel mode