Python Google Cloud PubSub TimeOut Exception

Question

I am busy working with PubSub using Google Cloud and I have implemented Google PubSub Subscribers and the stack is working as expected however I sometimes get an issue whereby a timeout occurs on my callback and then the subscriber thread seems to hang as seen below:

DEBUG 2019-08-14 09:49:13,130 google.cloud.pubsub_v1.subscriber.policy.thread.dispatch_callback:287- Handling 1 batched requests
DEBUG 2019-08-14 09:49:15,441 google.cloud.pubsub_v1.subscriber.policy.base.maintain_leases:347- The current p99 value is 10 seconds.
DEBUG 2019-08-14 09:49:15,442 google.cloud.pubsub_v1.subscriber.policy.base.maintain_leases:397- Snoozing lease management for 2.925854 seconds.
ERROR 2019-08-14 09:49:16,315 models.subscribe:104- Subscriber Timeout occurred Timed out waiting for result.

This is specifically the error:

 Timed out waiting for result.

Which is described in the google cloud docs here .

My actual subscriber code and exception handling is as follows:

    protocol = self.get_subscriber_protocol()

    logger.info("Starting subscriber %s on topic %s", os.environ.get('PUBSUB_CLIENT_ID'), topic)

    future = protocol.subscribe(topic,
                                callback=callback,
                                always_raise=False,
                                create_topic=True,
                                exception_handler=exception_handler)

    try:
        future.result(timeout=120)
    except TimeoutError as te:
        logger.error("Subscriber Timeout occurred {}".format(te))
    except Exception as e:
        logger.error(e)

From what I understand, if a Time out occurs then the subscriber should log a message and discard that message however instead it seems to block the thread.

I was just wondering if anyone has experienced this and what would be the best approach to handle this case?

Thanks!

Answer 1

When you call protocol.subscribe() , are you retrieving the future returned by subscribe() in the Pub/Sub Python client library? If so, you should not call result() with a timeout on that future. https://googleapis.github.io/google-cloud-python/latest/pubsub/subscriber/api/client.html#google.cloud.pubsub_v1.subscriber.client.Client.subscribe