Google Cloud PubSub 在异步拉取消息时抛出 504 Deadline Exceeded 错误

Question

I have a service that subscribes to a PubSub topic and uses the asynchronous pull. After an idle of 10 minutes and not receiving any messages, PubSub throws an 504 Deadline exceeded error.

Error always occurs after about 10 minutes. Every similar issue I found was related to synchronous pull, not the asynchronous pull I use.

Error message:

INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-terminating stream error 504 Deadline Exceeded                                                                                                           
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded                                                                                                                 
INFO:google.api_core.bidi:Re-established stream                                                                         
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-terminating stream error 504 Deadline Exceeded                                                                                                             
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded

Subscriber Class:

from google.cloud import pubsub_v1
class Subscriber():

    def __init__(self):
        self.subscriber = pubsub_v1.SubscriberClient()
        self.project_id = "my-project-id"
        self.subscription_id = "SUBSCRIPTION_MAIL_RECIEVE"
        self.subscription_path = self.subscriber.subscription_path(self.project_id,
                                                                   self.subscription_id)

    def subscribe(self, callback):
        """
        Parameters:
            callback (callable): function to be called for incoming messages on given topic
        """
        streaming_pull_future = self.subscriber.subscribe(self.subscription_path,
                                                          callback=callback)
        return streaming_pull_future

Listening for messages:

subscriber = Subscriber()
pull_future = subscriber.subscribe(my_callback_function(message))

with subscriber.subscriber:
        try:
            print("Waiting for messages")
            pull_future.result()
        except TimeoutError:
            pull_future.cancel()

Answer 1

This is the normal behaviour and these messages can be ignored as your stream is getting re-established. If in case of the stream is not getting re-established and heartbeat is not sent, then probably you need to upgrade your google core API and google cloud Pub/Sub to latest versions.

pip install google-cloud-pubsub --upgrade

pip install google-api-core --upgrade

You can also filter these error messages as suggested in link https://github.com/googleapis/google-cloud-python/issues/5800