Multiple node.js message receivers for a redis queue

Question

Trying to implement queue processor(s) in node.js application using rsmq-worker ( https://github.com/mpneuried/rsmq-worker ). Have to run multiple workers to process messages in parallel.

How do we setup multiple workers?

Exploring on using - https://github.com/Automattic/kue#parallel-processing-with-cluster

Answer 1

Simply create multiple instances of the RSMQWorker pointing to the same queue.

Example:

var workers = [];
for (let i = 0; i < NUMBER_OF_WORKERS; i++) {
        workers[i] = createWorker(i);
}

function createWorker(i) {
    const worker = new RSMQWorker(QUEUE_NAME, {...});
    worker.on("message", processMessage)
    return worker;
}

Then you can use the auto-start option of the rsmq-worker or start them manually:

for (let i = 0; i < workers.length; i++) {
    workers[i].start();
}

With this approach you will be processing multiple messages in parallel using one single Node.js instance. It improves the performance if your message processing logic involves some I/O.

For an additional level of parallelism, you can run the above code in multiple instances of Node.js, as mentioned in other answer.

Answer 2

You just launch multiple worker processes and they will start to process messages in parallel. RSMQ states that it guarantees that messages will be delivered only once so only one worker will process a single message.

Take a look at RSMQ readme, https://github.com/smrchy/rsmq . Pay attention to:

Guaranteed delivery of a message to exactly one recipient within a messages visibility timeout.

Please have a look at the createQueue and receiveMessage methods described below for optional parameters like visibility timeout and delay.