Pub/Sub 测试：即使通过了 ack_deadline，客户端也会收到消息

Question

I'm testing Cloud Pub/Sub. According to google documentation, ack_deadline of a pull substription can be set between 10s-600s ie. msg will be redelivered by Pubsub if ack_deadline is passed.

I'm processing the pubsub message in subscriber client before ack-ing the msg. This processing time can take ~ 700s which exceeds the max limit of 600s.

reproduction:

1. create a topic and subscription (by default Acknowledgement deadline is set to 10s)
2. run subscriber code (which ack the messages) see below
3. publish some msg on the topic from Web UI

subscriber code:

import time
import datetime
from concurrent.futures import TimeoutError
from google.cloud import pubsub_v1


project_id = "my-project"
subscription_id = "test-sub"
def sub():

    subscriber = pubsub_v1.SubscriberClient()
    subscription_path = subscriber.subscription_path(project_id, subscription_id)

    def callback(message: pubsub_v1.subscriber.message.Message) -> None: 
        # My processing code, which takes 700s
        time.sleep(700) # sleep function to demonstrate processing 
        print(f"Received {message}."+ str(datetime.datetime.now()) )
        message.ack()
        print("msg acked")

    streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
    print(f"Listening for messages on {subscription_path}..\n")

    try:
        streaming_pull_future.result()
    except:  
        streaming_pull_future.cancel()  # Trigger the shutdown.
        streaming_pull_future.result()  # Block until the shutdown is complete.

    subscriber.close()
       
if __name__ == "__main__":
    sub()

Even if the ack_deadline is reached, the message is getting acked which is weird. According to my understanding, pubsub should redeliver the message again and eventually go this code will go into an infinite loop.

am I missing something here?

Answer 1

That's correct. There are several solutions with their tradeoff

• Ack immediately the message and process it. The problem is: if you have an outage on your system, you lost the message
• Save the message ID state in a database (firestore for instance).
  • If the message ID is new, start the processing; at the end of the processing update the message ID status in the database
  • If the message ID already exists, sleep a while (about 90s), check the status of the message ID in the database. If DONE, ack the message. If not, sleep again (max 6 time. Then NACK and start again that process. To break the loop, repeat the process until the message timestamp is above 1h)
• Save the message in database, ack the message, and start the processing. In case of outage, at the start, check the not yet done messages and restart the process for each of them. At the end of the process, mark them as DONE.

You can also imagine other pattern. nothing is real perfect, depends on your needs.

Answer 2

The reason that the message is getting acked and not getting redelivered even after the ack deadline specified in the subscription is reached is that the Pub/Sub client libraries internally extend ack deadlines up to a time specified when instantiating the subscriber client. By default, the time is 1 hour. You can change this amount of time by changing the max_lease_duration parameter in the FlowControl object (search for "FlowControl" in the Types page ) passed into the subscribe method .