Google Cloud PubSub：不發送/接收來自 Cloud Functions 的所有消息

Question

摘要：我的客戶端代碼通過將消息發布到 Pub/Sub 主題來觸發 861 后台 Google Cloud Function。 每個 Cloud Function 執行一項任務，將結果上傳到 Google Storage，並將消息發布到客戶端代碼正在偵聽的另一個 Pub/Sub 主題。 盡管執行了所有 Cloud Functions（通過 Google Storage 中的結果數量進行驗證），但客戶端代碼並未收到所有消息。

服務器端：我有一個后台 Google Cloud Function，每次將消息發布到 TRIGGER Pub/Sub 主題時都會觸發該功能。 消息數據的自定義屬性充當函數參數，具體取決於函數執行特定任務。 然后將結果上傳到 Google Storage 中的存儲桶，並向 RESULTS Pub/Sub 主題（與用於觸發此功能的主題不同）發布一條消息（帶有 taskID 和執行時間詳細信息）。

客戶端：我需要執行 861 個不同的任務，這需要使用 861 個稍微不同的輸入調用 Cloud Function。 這些任務是相似的，Cloud Function 執行它們需要 20 秒到 2 分鍾（中位數約為 1 分鍾）。 我為此創建了一個從 Google Cloud Shell（或本地機器 shell）運行的 python 腳本。 客戶端 python 腳本將 861 條消息發布到 TRIGGER Pub/Sub 主題，該主題同時觸發了盡可能多的 Cloud Functions，每個都被傳遞了一個唯一的 taskID 范圍 [0, 860]。 然后，客戶端 python 腳本以“同步拉取”方式輪詢 RESULTS Pub/Sub 主題以獲取任何消息。 Cloud Function 執行任務后，使用唯一的 taskID 和時間詳細信息將消息發布到 RESULTS Pub/Sub 主題。 客戶端使用這個唯一的 taskID 來識別消息來自哪個任務。 它還有助於識別被丟棄的重復消息。

基本步驟：

1. 客戶端 python 腳本向 TRIGGER Pub/Sub 主題發布 861 條消息（每條消息都有唯一的 taskID）並等待來自 Cloud Function 的結果消息。
2. 調用了 861 個不同的 Cloud Functions，每個函數執行一個任務，將結果上傳到 Google Storage，並將消息（帶有 taskID 和執行時間詳細信息）發布到 RESULTS Pub/Sub 主題。
3. 客戶端同步獲取所有消息並將任務標記為完成。

問題：當客戶端輪詢來自 RESULTS Pub/Sub 主題的消息時，我沒有收到所有 taskID 的消息。 我確信 Cloud Function 被正確調用和執行（我在 Google Storage 存儲桶中有 861 個結果）。 我重復了很多次，每次都會發生。 奇怪的是，丟失的 taskID 的數量每次都會改變，並且不同的 taskID 在不同的運行中丟失。 我還跟蹤收到的重復 taskID 的數量。 表中給出了 5 次獨立運行的接收、丟失和重復的唯一任務 ID 的數量。

SN   # of Tasks  Received  Missing  Repeated
1     861          860      1        25
2     861          840      21       3
3     861          851      10       1
4     861          837      24       3
5     861          856      5        1

我不確定這個問題可能來自哪里。 鑒於數字的隨機性質以及丟失的 taskID，我懷疑 Pub/Sub 至少一次交付邏輯中存在一些錯誤。 如果在 Cloud Functions 中，我睡了幾秒鍾而不是執行任務，例如使用 time.sleep(5)，那么一切正常（我在客戶端收到所有 861 taskID）。

重現此問題的代碼。

在下文中， main.py和requirements.txt被部署為 Google Cloud Function，而client.py是客戶端代碼。 以python client.py 100形式運行具有 100 個並發任務的客戶端，重復 5 次。 每次丟失不同數量的 taskID。

requirements.txt

google-cloud-pubsub

main.py

"""
This file is deployed as Google Cloud Function. This function starts,
sleeps for some seconds and pulishes back the taskID.

Deloyment:
    gcloud functions deploy gcf_run --runtime python37 --trigger-topic <TRIGGER_TOPIC> --memory=128MB --timeout=300s
"""

import time
from random import randint
from google.cloud import pubsub_v1

# Global variables
project_id = "<Your Google Cloud Project ID>"  # Your Google Cloud Project ID
topic_name = "<RESULTS_TOPIC>"  # Your Pub/Sub topic name


def gcf_run(data, context):
    """Background Cloud Function to be triggered by Pub/Sub.
    Args:
         data (dict): The dictionary with data specific to this type of event.
         context (google.cloud.functions.Context): The Cloud Functions event
         metadata.
    """

    # Message should contain taskID (in addition to the data)
    if 'attributes' in data:
        attributes = data['attributes']
        if 'taskID' in attributes:
            taskID = attributes['taskID']
        else:
            print('taskID missing!')
            return
    else:
        print('attributes missing!')
        return

    # Sleep for a random time beteen 30 seconds to 1.5 minutes
    print("Start execution for {}".format(taskID))
    sleep_time = randint(30, 90)  # sleep for this many seconds
    time.sleep(sleep_time)  # sleep for few seconds

    # Marks this task complete by publishing a message to Pub/Sub.
    data = u'Message number {}'.format(taskID)
    data = data.encode('utf-8')  # Data must be a bytestring
    publisher = pubsub_v1.PublisherClient()
    topic_path = publisher.topic_path(project_id, topic_name)
    publisher.publish(topic_path, data=data, taskID=taskID)

    return

client.py

"""
The client code creates the given number of tasks and publishes to Pub/Sub,
which in turn calls the Google Cloud Functions concurrently.
Run:
    python client.py 100
"""

from __future__ import print_function
import sys
import time
from google.cloud import pubsub_v1

# Global variables
project_id = "<Google Cloud Project ID>" # Google Cloud Project ID
topic_name = "<TRIGGER_TOPIC>"    # Pub/Sub topic name to publish
subscription_name = "<subscriber to RESULTS_TOPIC>"  # Pub/Sub subscription name
num_experiments = 5  # number of times to repeat the experiment
time_between_exp = 120.0 # number of seconds between experiments

# Initialize the Publisher (to send commands that invoke Cloud Functions)
# as well as Subscriber (to receive results written by the Cloud Functions)
# Configure the batch to publish as soon as there is one kilobyte
# of data or one second has passed.
batch_settings = pubsub_v1.types.BatchSettings(
    max_bytes=1024,  # One kilobyte
    max_latency=1,   # One second
)
publisher = pubsub_v1.PublisherClient(batch_settings)
topic_path = publisher.topic_path(project_id, topic_name)

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    project_id, subscription_name)


class Task:
    """
    A task which will execute the Cloud Function once.

    Attributes:
        taskID (int)       : A unique number given to a task (starting from 0).
        complete (boolean) : Flag to indicate if this task has completed.
    """
    def __init__(self, taskID):
        self.taskID = taskID
        self.complete = False

    def start(self):
        """
        Start the execution of Cloud Function by publishing a message with
        taskID to the Pub/Sub topic.
        """
        data = u'Message number {}'.format(self.taskID)
        data = data.encode('utf-8')  # Data must be a bytestring
        publisher.publish(topic_path, data=data, taskID=str(self.taskID))

    def end(self):
        """
        Mark the end of this task.
            Returns (boolean):
                True if normal, False if task was already marked before.
        """
        # If this task was not complete, mark it as completed
        if not self.complete:
            self.complete = True
            return True

        return False
    # [END of Task Class]


def createTasks(num_tasks):
    """
    Create a list of tasks and return it.
        Args:
            num_tasks (int) : Number of tasks (Cloud Function calls)
        Returns (list):
            A list of tasks.
    """
    all_tasks = list()
    for taskID in range(0, num_tasks):
        all_tasks.append(Task(taskID=taskID))

    return all_tasks


def receiveResults(all_tasks):
    """
    Receives messages from the Pub/Sub subscription. I am using a blocking
    Synchronous Pull instead of the usual asynchronous pull with a callback
    funtion as I rely on a polling pattern to retrieve messages.
    See: https://cloud.google.com/pubsub/docs/pull
        Args:
            all_tasks (list) : List of all tasks.
    """
    num_tasks = len(all_tasks)
    total_msg_received = 0  # track the number of messages received
    NUM_MESSAGES = 10  # maximum number of messages to pull synchronously
    TIMEOUT = 600.0    # number of seconds to wait for response (10 minutes)

    # Keep track of elapsed time and exit if > TIMEOUT
    __MyFuncStartTime = time.time()
    __MyFuncElapsedTime = 0.0

    print('Listening for messages on {}'.format(subscription_path))
    while (total_msg_received < num_tasks) and (__MyFuncElapsedTime < TIMEOUT):
        # The subscriber pulls a specific number of messages.
        response = subscriber.pull(subscription_path,
            max_messages=NUM_MESSAGES, timeout=TIMEOUT, retry=None)
        ack_ids = []

        # Keep track of all received messages
        for received_message in response.received_messages:
            if received_message.message.attributes:
                attributes = received_message.message.attributes
                taskID = int(attributes['taskID'])
                if all_tasks[taskID].end():
                    # increment count only if task completes the first time
                    # if False, we received a duplicate message
                    total_msg_received += 1
                #     print("Received taskID = {} ({} of {})".format(
                #         taskID, total_msg_received, num_tasks))
                # else:
                #     print('REPEATED: taskID {} was already marked'.format(taskID))
            else:
                print('attributes missing!')

            ack_ids.append(received_message.ack_id)

        # Acknowledges the received messages so they will not be sent again.
        if ack_ids:
            subscriber.acknowledge(subscription_path, ack_ids)

        time.sleep(0.2)  # Wait 200 ms before polling again
        __MyFuncElapsedTime = time.time() - __MyFuncStartTime
        # print("{} s elapsed. Listening again.".format(__MyFuncElapsedTime))

    # if total_msg_received != num_tasks, function exit due to timeout
    if total_msg_received != num_tasks:
        print("WARNING: *** Receiver timed out! ***")
    print("Received {} messages out of {}. Done.".format(
        total_msg_received, num_tasks))


def main(num_tasks):
    """
    Main execution point of the program
    """

    for experiment_num in range(1, num_experiments + 1):
        print("Starting experiment {} of {} with {} tasks".format(
            experiment_num, num_experiments, num_tasks))
        # Create all tasks and start them
        all_tasks = createTasks(num_tasks)
        for task in all_tasks:     # Start all tasks
            task.start()
        print("Published {} taskIDs".format(num_tasks))

        receiveResults(all_tasks)  # Receive message from Pub/Sub subscription

        print("Waiting {} seconds\n\n".format(time_between_exp))
        time.sleep(time_between_exp)  # sleep between experiments


if __name__ == "__main__":
    if(len(sys.argv) != 2):
        print("usage: python client.py  <num_tasks>")
        print("    num_tasks: Number of concurrent Cloud Function calls")
        sys.exit()

    num_tasks = int(sys.argv[1])
    main(num_tasks)

Answer 1

在您的雲函數中，在這一行中：

發布者.發布（主題路徑，數據=數據，任務ID=任務ID）

您不是在等待publisher.publish 返回的未來。 這意味着您不能保證當您從gcf_run函數結束時發布到主題上確實發生了，但是 TRIGGER 主題雲函數訂閱上的消息無論如何都會被確認。

相反，要等到發布發生以終止雲功能，這應該是：

publisher.publish(topic_path, data=data, taskID=taskID).result()

您還應該避免在每次函數調用時啟動和拆除發布者客戶端，而是將客戶端作為全局變量。