[英]GCP message stays in the Pub/Sub after acknowledge
我將 Pub/Sub 訂閱邏輯包裝在一個訂閱方法中,該方法在每個訂閱的服務初始化期間調用一次:
def subscribe(self,
callback: typing.Callable,
subscription_name: str,
topic_name: str,
project_name: str = None) -> typing.Optional[SubscriberClient]:
"""Subscribes to Pub/Sub topic and return subscriber client
:param callback: subscription callback method
:param subscription_name: name of the subscription
:param topic_name: name of the topic
:param project_name: optional project name. Uses default project if not set
:return: subscriber client or None if testing
"""
project = project_name if project_name else self.pubsub_project_id
self.logger.info('Subscribing to project `{}`, topic `{}`'.format(project, topic_name))
project_path = self.pubsub_subscriber.project_path(project)
topic_path = self.pubsub_subscriber.topic_path(project, topic_name)
subscription_path = self.pubsub_subscriber.subscription_path(project, subscription_name)
# check if there is an existing subscription, if not, create it
if subscription_path not in [s.name for s in self.pubsub_subscriber.list_subscriptions(project_path)]:
self.logger.info('Creating new subscription `{}`, topic `{}`'.format(subscription_name, topic_name))
self.pubsub_subscriber.create_subscription(subscription_path, topic_path)
# subscribe to the topic
self.pubsub_subscriber.subscribe(
subscription_path, callback=callback,
scheduler=self.thread_scheduler
)
return self.pubsub_subscriber
這個方法是這樣調用的:
self.subscribe_client = self.subscribe(
callback=self.pubsub_callback,
subscription_name='subscription_topic',
topic_name='topic'
)
回調方法做了很多事情,發送 2 封電子郵件,然后確認消息
def pubsub_callback(self, data: gcloud_pubsub_subscriber.Message):
self.logger.debug('Processing pub sub message')
try:
self.do_something_with_message(data)
self.logger.debug('Acknowledging the message')
data.ack()
self.logger.debug('Acknowledged')
return
except:
self.logger.warning({
"message": "Failed to process Pub/Sub message",
"request_size": data.size,
"data": data.data
}, exc_info=True)
self.logger.debug('Acknowledging the message 2')
data.ack()
當我向訂閱運行推送某些內容時,回調運行,打印所有調試消息,包括Acknowledged
。 但是,消息保留在 Pub/Sub 中,回調被再次調用,並且每次重試后都需要指數級的時間。 問題是,即使在調用ack
之后,什么可能導致消息保留在 pub/sub 中?
我有幾個這樣的訂閱,它們都按預期工作。 截止日期不是一個選項,回調幾乎立即完成,無論如何我都玩了確認截止日期,沒有任何幫助。
當我嘗試從連接到該 pub-sub 的本地運行的應用程序處理這些消息時,它完成得很好,並且確認按預期將消息從隊列中取出。
有任何想法嗎?
確認是 Pub/Sub 中的最大努力,因此重新傳遞消息是可能的,但不常見。
如果您一直收到重復消息,則可能是由於相同消息內容的重復發布。 就 Pub/Sub 而言,這些是不同的消息,將被分配不同的消息 ID。 檢查 Pub/Sub 提供的消息 ID,以確保您實際上多次收到相同的消息。
使用流式拉取處理大量積壓的小消息有一個極端情況(這是 Python 客戶端庫使用的)。 如果您正在運行多個訂閱同一訂閱的客戶端,則此邊緣情況可能是相關的。
您還可以檢查訂閱的Stackdriver 指標以查看:
subscription/ack_message_count
)subscription/backlog_bytes
積壓字節)subscription/streaming_pull_ack_message_operation_count
response_code != "success"
)如果您沒有錯過確認截止日期並且您的積壓工作保持穩定,您應該聯系 Google Cloud 支持並提供您的項目名稱、訂閱名稱和重復消息 ID 的示例。 他們將能夠調查為什么會發生這些重復。
我做了一些額外的測試,終於找到了問題所在。
TL;DR:我對所有訂閱都使用相同的google.cloud.pubsub_v1.subscriber.scheduler.ThreadScheduler
。
這是我用來測試它的代碼片段。 這是損壞的版本:
服務器.py
import concurrent.futures.thread
import os
import time
from google.api_core.exceptions import AlreadyExists
from google.cloud import pubsub_v1
from google.cloud.pubsub_v1.subscriber.scheduler import ThreadScheduler
def create_subscription(project_id, topic_name, subscription_name):
"""Create a new pull subscription on the given topic."""
subscriber = pubsub_v1.SubscriberClient()
topic_path = subscriber.topic_path(project_id, topic_name)
subscription_path = subscriber.subscription_path(
project_id, subscription_name)
subscription = subscriber.create_subscription(
subscription_path, topic_path)
print('Subscription created: {}'.format(subscription))
def receive_messages(project_id, subscription_name, t_scheduler):
"""Receives messages from a pull subscription."""
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project_id, subscription_name)
def callback(message):
print('Received message: {}'.format(message.data))
message.ack()
subscriber.subscribe(subscription_path, callback=callback, scheduler=t_scheduler)
print('Listening for messages on {}'.format(subscription_path))
project_id = os.getenv("PUBSUB_PROJECT_ID")
publisher = pubsub_v1.PublisherClient()
project_path = publisher.project_path(project_id)
# Create both topics
try:
topics = [topic.name.split('/')[-1] for topic in publisher.list_topics(project_path)]
if 'topic_a' not in topics:
publisher.create_topic(publisher.topic_path(project_id, 'topic_a'))
if 'topic_b' not in topics:
publisher.create_topic(publisher.topic_path(project_id, 'topic_b'))
except AlreadyExists:
print('Topics already exists')
# Create subscriptions on both topics
sub_client = pubsub_v1.SubscriberClient()
project_path = sub_client.project_path(project_id)
try:
subs = [sub.name.split('/')[-1] for sub in sub_client.list_subscriptions(project_path)]
if 'topic_a_sub' not in subs:
create_subscription(project_id, 'topic_a', 'topic_a_sub')
if 'topic_b_sub' not in subs:
create_subscription(project_id, 'topic_b', 'topic_b_sub')
except AlreadyExists:
print('Subscriptions already exists')
scheduler = ThreadScheduler(concurrent.futures.thread.ThreadPoolExecutor(10))
receive_messages(project_id, 'topic_a_sub', scheduler)
receive_messages(project_id, 'topic_b_sub', scheduler)
while True:
time.sleep(60)
客戶端.py
import datetime
import os
import random
import sys
from time import sleep
from google.cloud import pubsub_v1
def publish_messages(pid, topic_name):
"""Publishes multiple messages to a Pub/Sub topic."""
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(pid, topic_name)
for n in range(1, 10):
data = '[{} - {}] Message number {}'.format(datetime.datetime.now().isoformat(), topic_name, n)
data = data.encode('utf-8')
publisher.publish(topic_path, data=data)
sleep(random.randint(10, 50) / 10.0)
project_id = os.getenv("PUBSUB_PROJECT_ID")
publish_messages(project_id, sys.argv[1])
我連接到雲發布/訂閱,服務器創建主題和訂閱。 然后我為這兩個主題並行運行了多次客戶端腳本。 過了一會兒,一旦我更改服務器代碼以在receive_messages
scope 中實例化新的線程調度程序,服務器清理了兩個主題並按預期運行。
令人困惑的是,無論哪種情況,服務器都會打印出所有消息的接收消息。
我將把這個發布到https://github.com/googleapis/google-cloud-python/issues
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.