Node.js Google PubSub Subscriber does not receive SOME messages

Question

Summary:

I have a chat functionality in my Node.js app and want to send messages to the clients via socket.io. I trigger the emit to the client via PubSub. Now, when running the PubSub Subscription everything works (ie prints out messages) in roughly 70% of the cases, but would sometimes just stop and not do anything (in particular it would also not print out an error). I can confirm that the missing 30% (messages) are being published to the topic though, as a different subscription to the same topic receives the messages.

Any help with debugging this would be highly appreciated.

Details

This is my Stack:

• Node.js
• socket.io
• express.js
• passport.js
• MongoDB
• react.js

Process:

• Anna sends a message in chat (this writes to the database and also publishes to PubSub topic "messages")
• Node.js express app runs a subscription and would then based on the message content emit to whoever else should receive the message.
• BoB who is on the same channel as Anna would, in this case, receive the message.

Why do I not directly emit from Anna to Bob? The reason being that I want to have an AppEngine in between the messages and potentially add some logic there, this seemed a good way doing it.

Subscription

const pubSubClient = require('./client');

const errorHandler = function(error) {
    console.error(`ERROR: ${error}`);
    throw error;
};

module.exports = listenForMessages =(subscriptionName="messageReceiver",io) => {
    const subscription = pubSubClient.subscription(subscriptionName);

    // Listen for new messages until timeout is hit
    subscription.on("message", (message) => {
        console.log(`Received message ${message.id}:`);
        const parsedMessage = JSON.parse(message.data)
        parsedMessage._id = message.id
        console.log(parsedMessage)

        if (parsedMessage.to === "admin") {
            io.to(`admin:${parsedMessage.from}`).emit("NewMessage", parsedMessage);
        } else {
            io.to(`${parsedMessage.to}`).emit("NewMessage", parsedMessage);
        }
        message.ack();
    });

    subscription.on('error', errorHandler);
}

Server.js

...
const listenForMessages = require("./message_processing/listen");
listenForMessages("messageReceiver", io);

Sample console output

The following console output was generated by running the app locally with two browsers [one in incognito] chatting with each other. It can be seen that only the very last message was actually picked up by the listener (and printed out). Funnily enough, due to the async nature of the calls, the printout of the received message came before the log that the message was sent (ie latency surely can't be a problem here).

[0] went into deserialize user
[0] Message 875007020424601 published.
[0] went into deserialize user
[0] Message 875006704834317 published.
[0] went into deserialize user
[0] Message 875006583857400 published.
[0] went into deserialize user
[0] Message 875006520104287 published.
[0] went into deserialize user
[0] Message 875006699141463 published.
[0] went into deserialize user
[0] Received message 875006881073134:
[0] {
[0]   from: '5e949f73aeed81beefaf6daa',
[0]   to: 'admin',
[0]   content: 'i6',
[0]   seenByUser: true,
[0]   type: 'message',
[0]   createdByUser: true,
[0]   createdAt: '2020-04-20T07:44:54.280Z',
[0]   _id: '875006881073134'
[0] }
[0] Message 875006881073134 published.

In some other cases, earlier messages work and then the listener seems to stop.

Answer 1

There are a couple of things you could do to check what is happening:

1. Go to the topic page , select the topic to see the details, and look at the publish rate. Ensure that the messages you think are being published are actually being published successfully as far as Pub/Sub is concerned. If publishes are failing, it is possible they could be delivered to one of your subscribers and not the other.
2. Go to the subscription page , select the subscription to see the details, and look at the "Unacked message count" and "Oldest unacked message age" graphs. If these are nonzero, then that means there are messages not getting delivered to your subscriber. If they are zero, then that means the messages are getting delivered to and acknowledged by your subscriber.

If the number of unacked messages is zero, then the likely cause is a rogue processes acting as a subscriber on the subscription receiving the messages. Perhaps a previous instance of your service is still running unexpectedly? Or possibly another task that should use a different subscription is using the same subscription.

Another thing to be aware of is that subscribers will only receive messages on subscriptions that were created before messages were published. Therefore, if you started up the publisher and published some messages and then created the subscription, say at the time when the subscriber was started up, then the subscriber will not receive those earlier messages.