Node.js application listens to message queue and add messages to redis asynchronously

Question

I am developing a node.js component that listens to a message queue (ActiveMQ) and adds the received messages to redis in batches (must be 20 per batch).

There is no problem when number of messages received from ActiveMQ is 10 per second or less.

My problem is that messages are added to the queue every 4 milliseconds. Which causes the number of records added to the batch to be sometimes more than 20 per batch.

const stompit = require('stompit');
var uuid = require('node-uuid');
var Redis = require('ioredis');

var redis = new Redis();
var pipeline = redis.pipeline();
var batchCounter = 0;

stompit.connect({ host: 'localhost', port: 61613 }, function(err1, client) {

client.subscribe({ destination: 'MyQueue' }, function(err2, msg) {
    msg.readString('UTF-8', function(err3, body) {

        if (batchCounter >= 20){
            pipeline.exec(function(err4, results) {
                pipeline = redis.pipeline();
                batchCounter = 0;
                console.log(results);
            });
        }

        batchCounter++;
        pipeline.set(uuid.v1(), JSON.stringify(body));
        //client.disconnect();
      });
  });
  });

How can fix this problem? Thank you

Answer 1

Try resetting the pipeline before calling the .exec method, which I assume is an asynchronous method. Because .exec runs at some point in the future, the increment and pipeline.set can run before it.

The following code saves the current pipeline and creates a new one synchronously before the .exec

if (batchCounter >= 20){
    let fullpipeline = pipeline;
    pipeline = redis.pipeline();
    batchCounter = 0; 
    fullpipeline.exec(function(err4, results) {
        console.log(err4, results);
    });
}

Then new messages should then only be appended to the new pipeline.

Answer 2

I ended up controlling number of records per batch using flags. I believe there is another possible, probably more efficient, fix to this which is controlling the process of reading from ActiveMQ. But, in this situation, flags did the job for me. Complete code using flags is in the following link:

https://github.com/TamerB/ToActiveMQToRedis/blob/master/consumer/app.js