带有aws-sdk的Amazon SQS receiveMessage Stall

Question

I'm using the aws-sdk node module with the (as far as I can tell) approved way to poll for messages.

Which basically sums up to:

        sqs.receiveMessage({
            QueueUrl: queueUrl,
            MaxNumberOfMessages: 10,
            WaitTimeSeconds: 20
        }, function(err, data) {
            if (err) {
                logger.fatal('Error on Message Recieve');
                logger.fatal(err);
            } else {
                // all good
                if (undefined === data.Messages) {
                    logger.info('No Messages Object');
                } else if (data.Messages.length > 0) {
                    logger.info('Messages Count: ' + data.Messages.length);

                    var delete_batch = new Array();
                    for (var x=0;x<data.Messages.length;x++) {
                        // process
                        receiveMessage(data.Messages[x]);

                        // flag to delete

                        var pck = new Array();
                        pck['Id'] = data.Messages[x].MessageId;
                        pck['ReceiptHandle'] = data.Messages[x].ReceiptHandle;

                        delete_batch.push(pck);
                    }

                    if (delete_batch.length > 0) {
                        logger.info('Calling Delete');
                        sqs.deleteMessageBatch({
                            Entries: delete_batch,
                            QueueUrl: queueUrl
                        }, function(err, data) {
                            if (err) {
                                logger.fatal('Failed to delete messages');
                                logger.fatal(err);
                            } else {
                                logger.debug('Deleted recieved ok');
                            }
                        });
                    }
                } else {
                    logger.info('No Messages Count');
                }
            }
        });

receiveMessage is my "do stuff with collected messages if I have enough collected messages" function

Occasionally, my script is stalling because I don't get a response for Amazon at all, say for example there are no messages in the queue to consume and instead of hitting the WaitTimeSeconds and sending a "no messages object", the callback isn't called.

(I'm writing this up to Amazon Weirdness)

What I'm asking is whats the best way to detect and deal with this, as I have some code in place to stop concurrent calls to receiveMessage.

The suggested answer here: Nodejs sqs queue processor also has code that prevents concurrent message request queries (granted it's only fetching one message a time)

I do have the whole thing wrapped in

var running = false;
runMonitorJob = setInterval(function() {
    if (running) {
    } else {
        running = true;
        // call SQS.receive
    }
}, 500);

(With a running = false after the delete loop (not in it's callback))

My solution would be

watchdogTimeout = setTimeout(function() {
    running = false;
}, 30000);

But surely this would leave a pile of floating sqs.receive's lurking about and thus much memory over time?

(This job runs all the time, and I left it running on Friday, it stalled Saturday morning and hung till I manually restarted the job this morning)

Edit: I have seen cases where it hangs for ~5 minutes and then suddenly gets messages BUT with a wait time of 20 seconds it should throw a "no messages" after 20 seconds. So a WatchDog of ~10 minutes might be more practical (depending on the rest of ones business logic)

Edit: Yes Long Polling is already configured Queue Side.

Edit: This is under (latest) v2.3.9 of aws-sdk and NodeJS v4.4.4

Answer 1

I've been chasing this (or a similar) issue for a few days now and here's what I've noticed:

• The receiveMessage call does eventually return although only after 120 seconds

• Concurrent calls to receiveMessage are serialised by the AWS.SDK library so making multiple calls in parallel have no effect.

• The receiveMessage callback does not error - in fact after the 120 seconds have passed, it may contain messages.

What can be done about this? This sort of thing can happen for a number of reasons and some/many of these things can't necessarily be fixed. The answer is to run multiple services each calling receiveMessage and processing the messages as they come - SQS supports this. At any time, one of these services may hit this 120 second lag but the other services should be able to continue on as normal.

My particular problem is that I have some critical singleton services that can't afford 120 seconds of down time. For this I will look into either 1) use HTTP instead of SQS to push messages into my service or 2) spawn slave processes around each of the singletons to fetch the messages from SQS and push them into the service.

Answer 2

I also ran into this issue, but not when calling receiveMessage but sendMessage. I also saw hangups of exactly 120 seconds. I also saw it with a few other services, like Firehose.

That lead me to this line in the AWS SDK:

SQS Constructor

httpOptions:

• timeout [Integer] — Sets the socket to timeout after timeout milliseconds of inactivity on the socket. Defaults to two minutes (120000).

to implement a fix, I override the timeout for my SQS client that performs the sendMessage to timeout after 10 seconds, and another with 25 seconds for receiving (where I long poll for 20 seconds):

var sendClient    = new AWS.SQS({httpOptions:{timeout:10*1000}});
var receiveClient = new AWS.SQS({httpOptions:{timeout:25*1000}});

I've had this out in production for a week now and I've noticed that all of my SQS stalling issues have been eliminated.