如何通過轉發到死信主題來限制 Google Pub/Sub 交付嘗試？

Question

我正在嘗試使用死信主題（參見https://cloud.google.com/pubsub/docs/dead-letter-topics ）配置 Pub/Sub 訂閱，以限制消息的次數收到后會重新交付。 為此，我創建了以下示例程序：

package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    "os"
    "time"

    "cloud.google.com/go/pubsub"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)

const (
    topicID           = "my-topic"
    deadLetterTopicID = "my-dead-letter-topic"
    subscriptionID    = "my-subscription"
)

var (
    pubsubEmulatorHost string
    projectID          string
)

func main() {
    flag.StringVar(&pubsubEmulatorHost, "pubsubEmulatorHost", "", "Pub/Sub emulator host (e.g. localhost:8085)")
    flag.StringVar(&projectID, "projectID", "my-project", "Google Project ID")
    flag.Parse()

    if pubsubEmulatorHost != "" {
        os.Setenv("PUBSUB_EMULATOR_HOST", pubsubEmulatorHost)
        defer os.Unsetenv("PUBSUB_EMULATOR_HOST")
    }

    client, err := pubsub.NewClient(context.Background(), projectID)
    if err != nil {
        log.Fatalf("NewClient: %v", err)
    }

    topic, err := client.CreateTopic(context.Background(), topicID)
    if err != nil {
        if status.Code(err) == codes.AlreadyExists {
            topic = client.Topic(topicID)
            log.Printf("Topic %s already exists", topicID)
        } else {
            log.Fatalf("CreateTopic: %v", err)
        }
    }
    defer func() {
        topic.Stop()
        if err := topic.Delete(context.Background()); err != nil {
            log.Fatalf("Delete topic: %v", err)
        }
    }()

    deadLetterTopic, err := client.CreateTopic(context.Background(), deadLetterTopicID)
    if err != nil {
        if status.Code(err) == codes.AlreadyExists {
            deadLetterTopic = client.Topic(deadLetterTopicID)
            log.Printf("Topic %s already exists", deadLetterTopicID)
        } else {
            log.Fatalf("CreateTopic: %v", err)
        }
    }
    defer func() {
        deadLetterTopic.Stop()
        if err := deadLetterTopic.Delete(context.Background()); err != nil {
            log.Fatalf("Delete dead-letter topic: %v", err)
        }
    }()

    sub, err := client.CreateSubscription(context.Background(), subscriptionID, pubsub.SubscriptionConfig{
        Topic: topic,
        DeadLetterPolicy: &pubsub.DeadLetterPolicy{
            DeadLetterTopic:     fmt.Sprintf("projects/%s/topics/%s", projectID, deadLetterTopicID),
            MaxDeliveryAttempts: 5,
        },
    })
    if err != nil {
        log.Fatalf("CreateSubscription: %v", err)
    }
    defer func() {
        if err := sub.Delete(context.Background()); err != nil {
            log.Fatalf("Delete subscription: %v", err)
        }
    }()

    go func() {
        sub.Receive(context.Background(), func(ctx context.Context, msg *pubsub.Message) {
            log.Printf("Got message %q upon delivery attempt %d", msg.Data, msg.DeliveryAttempt)
            msg.Nack()
        })
    }()

    result := topic.Publish(context.Background(), &pubsub.Message{Data: []byte("Hello, world!")})
    messageID, err := result.Get(context.Background())
    if err != nil {
        log.Fatalf("Get message ID of publish call: %v", err)
    }
    log.Printf("Published message with ID %s", messageID)
    time.Sleep(20 * time.Second)
}

該腳本以兩種模式運行，一種使用真正的 Pub/Sub 項目（此處稱為my-project ），另一種通過設置PUBSUB_EMULATOR_HOST環境變量使用GCloud Pub/Sub 模擬器。 我希望，鑒於訂閱的DeadLetterPolicy將MaxDeliveryAttempts設置為 5，nack'd Pub/Sub 消息將被傳遞大約 5 次（文檔表明這是盡最大努力）。 但是，如果我在真正的 Pub/Sub 項目上運行腳本，我會得到以下 output：

> go run main.go
2020/06/22 23:59:37 Published message with ID 1294186248588871
2020/06/22 23:59:38 Got message "Hello, world!" upon delivery attempt 824637866440
2020/06/22 23:59:40 Got message "Hello, world!" upon delivery attempt 824634417896
2020/06/22 23:59:41 Got message "Hello, world!" upon delivery attempt 824634418592
2020/06/22 23:59:43 Got message "Hello, world!" upon delivery attempt 824637866928
2020/06/22 23:59:44 Got message "Hello, world!" upon delivery attempt 824638981864
2020/06/22 23:59:45 Got message "Hello, world!" upon delivery attempt 824640667960
2020/06/22 23:59:47 Got message "Hello, world!" upon delivery attempt 824634418712
2020/06/22 23:59:49 Got message "Hello, world!" upon delivery attempt 824638982160
2020/06/22 23:59:50 Got message "Hello, world!" upon delivery attempt 824640667760
2020/06/22 23:59:51 Got message "Hello, world!" upon delivery attempt 824634418000
2020/06/22 23:59:52 Got message "Hello, world!" upon delivery attempt 824633942168
2020/06/22 23:59:53 Got message "Hello, world!" upon delivery attempt 824633942712
2020/06/22 23:59:53 Got message "Hello, world!" upon delivery attempt 824640668296
2020/06/22 23:59:54 Got message "Hello, world!" upon delivery attempt 824637448352
2020/06/22 23:59:55 Got message "Hello, world!" upon delivery attempt 824633943336
2020/06/22 23:59:55 Got message "Hello, world!" upon delivery attempt 824633943448
2020/06/22 23:59:56 Got message "Hello, world!" upon delivery attempt 824633943560
2020/06/22 23:59:57 Got message "Hello, world!" upon delivery attempt 824638259688
2020/06/22 23:59:57 Got message "Hello, world!" upon delivery attempt 824637448752

換句話說，nack 的消息被傳遞了 19 次，與我預期的 5 次相去甚遠。 如果我使用 Pub/Sub 模擬器運行程序，我會發現交付嘗試始終為 0：

> go run main.go --pubsubEmulatorHost=localhost:8085
2020/06/23 00:00:54 Published message with ID 4
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:54 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:55 Got message "Hello, world!" upon delivery attempt 0
2020/06/23 00:00:56 Got message "Hello, world!" upon delivery attempt 0
...

為簡潔起見，此處 output 被截斷，但消息打印了大約 200 次（每秒 10 次，持續 20 秒），再次遠高於我預期的 5 次。

DeadLetterPolicy的MaxDeliveryAttempts字段是否不應該限制 nack 消息的傳遞嘗試次數？ 為什么DeliveryAttempt字段是一個如此奇怪的整數序列，而不是每次簡單地增加 1 的整數序列（參見https://pkg.go.dev/cloud.google.com/go/pubsub?tab=doc#Message ）？

Answer 1

為了進一步為社區做出貢獻，我根據我在評論部分中提到的內容發布了這個答案。

您描述的問題通常發生在您未提供所需權限（ 此處在死信隊列選項下）時，因此 PubSub 可以發布到您的死信主題或訂閱您的訂閱。 另外，我必須指出，如果寫入死信隊列主題失敗，PubSub 將繼續將消息傳遞給您的訂閱者。

為了提供必要的權限，您可以在 shell 環境中使用以下命令：

PUBSUB_SERVICE_ACCOUNT="service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"

gcloud pubsub topics add-iam-policy-binding <dead letter topic> \  --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\  --role='roles/pubsub.publisher'

gcloud pubsub subscriptions add-iam-policy-binding <subscription with dead letter queue> \  --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\  --role='roles/pubsub.subscriber'

另外，我想添加@Mahesh Gattani 評論，其中提到模擬器目前不支持死信主題。