Google Cloud PubSub messages not processed by callback

Question

I'm trying to use Google PubSub to pass and receive messages between two services. However, some of the messages sent seem to be randomly dropped, and are not processed by the subscriber's callback method.

When sending messages, about half the messages are processed by the callback method. For the other half, the callback method does not appear to be called at all (No info is logged). However, the messages still disappear from the topic, and are not resent.

The code used for starting the subscriber:

logger = logging.getLogger(LOGGER_NAME)
logger.info('Starting the pubsub subscriber')
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(GOOGLE_CLOUD_PROJECT, SUBSCRIPTION_NAME)
subscriber.subscribe(subscription_path, callback=callback)
while True:
    try:
        sleep(60)
    except Exception as e:
        // Log exception

The callback method:

def callback(message):
    logger = logging.getLogger(LOGGER_NAME)
    logger.info(f'Recieved callback with message: {message}', extra = {'callback_message': message}  )
    // Process message

The error appears to be on the subscriber side. The messages are sent from the publisher, and if a subscriber is not connected to the topic the messages do not disappear.

I've tried to use Flow Control to control the number of messages retrieved by the subscriber, but it does not appear to have any effect.

Can the messages be processed without invoking the callback method? Are there some other reasons why messages may disappear from the topic?

EDIT: Turns out another service was reading from the same subscription, processing the missing messages.

Answer 1

I know you found the answer to your problem, but I thought it would be worthwhile to list some useful steps for debugging this type of problem:

1. Check to ensure that the messages actually got published. When a publish succeeds, the response should include the ID of the message, eg, as the String yielded by the APIFuture in the Java Publish method.
2. Check to see if a backlog of messages is building up. You can view subscription/oldest_unacked_message_age and subscription/num_undelivered_messages via Stackdriver .
3. Check to see if you have flow control set on your subscriber that is preventing you from receiving all messages in a timely fashion. If you have flow control set and it is preventing delivery of all messages, you will likely see that the number of number of undelivered messages is increasing in Stackdriver.
4. Ensure you don't have any additional clients subscribing to messages for the same subscription. For example, perhaps you are using the gcloud tool to pull and look at messages. In this situation, you probably won't see the number of undelivered messages increasing in Stackdriver.

If after checking all of that you are unsure what is happening to your messages, it is best to contact support with the name of your project and subscription as well as the IDs of any messages you think were not delivered.