Google Pub/Sub 消息排序不起作用（或将延迟增加到 10 秒以上）？

Question

我正在尝试制作一个简化示例来演示如何使用 Google Pub/Sub 的消息排序功能（ https://cloud.google.com/pubsub/docs/ordering ）。 从这些文档中，为订阅启用消息排序后，

设置消息排序属性后，Pub/Sub 服务按照 Pub/Sub 服务接收消息的顺序下发具有相同 ordering key 的消息。 例如，如果发布者使用相同的排序键发送两条消息，则 Pub/Sub 服务首先传递最旧的消息。

我用它来编写以下示例：

package main

import (
    "context"
    "log"
    "time"

    "cloud.google.com/go/pubsub"
    uuid "github.com/satori/go.uuid"
)

func main() {
    client, err := pubsub.NewClient(context.Background(), "my-project")
    if err != nil {
        log.Fatalf("NewClient: %v", err)
    }

    topicID := "test-topic-" + uuid.NewV4().String()
    topic, err := client.CreateTopic(context.Background(), topicID)
    if err != nil {
        log.Fatalf("CreateTopic: %v", err)
    }
    defer topic.Delete(context.Background())

    subID := "test-subscription-" + uuid.NewV4().String()
    sub, err := client.CreateSubscription(context.Background(), subID, pubsub.SubscriptionConfig{
        Topic:                 topic,
        EnableMessageOrdering: true,
    })
    if err != nil {
        log.Fatalf("CreateSubscription: %v", err)
    }
    defer sub.Delete(context.Background())

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    messageReceived := make(chan struct{})
    go sub.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
        log.Printf("Received message with ordering key %s: %s", msg.OrderingKey, msg.Data)
        msg.Ack()
        messageReceived <- struct{}{}
    })

    topic.Publish(context.Background(), &pubsub.Message{Data: []byte("Dang1!"), OrderingKey: "foobar"})
    topic.Publish(context.Background(), &pubsub.Message{Data: []byte("Dang2!"), OrderingKey: "foobar"})

    for i := 0; i < 2; i++ {
        select {
        case <-messageReceived:
        case <-time.After(10 * time.Second):
            log.Fatal("Expected to receive a message, but timed out after 10 seconds.")
        }
    }
}

首先，我在没有在topic.Publish()调用中指定OrderingKey: "foobar"的情况下尝试了该程序。 这导致了以下 output：

> go run main.go
2020/08/10 21:40:34 Received message with ordering key : Dang2!
2020/08/10 21:40:34 Received message with ordering key : Dang1!

换句话说，消息的接收顺序与发布时的顺序不同，这在我的用例中是不可取的，我想通过指定OrderingKey来防止

但是，一旦我在发布调用中添加了OrderingKey ，程序在等待接收 Pub/Sub 消息 10 秒后就会超时：

> go run main.go
2020/08/10 21:44:36 Expected to receive a message, but timed out after 10 seconds.
exit status 1

我希望现在首先收到消息Dang1! 紧随其后的是Dang2! ，但我没有收到任何消息。 知道为什么这没有发生吗？

Answer 1

发布失败并出现以下错误： Failed to publish: Topic.EnableMessageOrdering=false, but an OrderingKey was set in Message. Please remove the OrderingKey or turn on Topic.EnableMessageOrdering Failed to publish: Topic.EnableMessageOrdering=false, but an OrderingKey was set in Message. Please remove the OrderingKey or turn on Topic.EnableMessageOrdering 。

如果您更改发布调用以检查错误，您可以看到这一点：

res1 := topic.Publish(context.Background(), &pubsub.Message{Data: []byte("Dang1!"), OrderingKey: "foobar"})
res2 := topic.Publish(context.Background(), &pubsub.Message{Data: []byte("Dang2!"), OrderingKey: "foobar"})

_, err = res1.Get(ctx)
if err != nil {
    fmt.Printf("Failed to publish: %v", err)
    return
}

_, err = res2.Get(ctx)
if err != nil {
    fmt.Printf("Failed to publish: %v", err)
    return
}

要修复它，请添加一行以启用主题的消息排序。 您的主题创建如下：

topic, err := client.CreateTopic(context.Background(), topicID)
if err != nil {
    log.Fatalf("CreateTopic: %v", err)
}
topic.EnableMessageOrdering = true
defer topic.Delete(context.Background())

Answer 2

我独立想出了与 Kamal 相同的解决方案，只是想分享完整的修改后的实现：

package main

import (
    "context"
    "flag"
    "log"
    "time"

    "cloud.google.com/go/pubsub"
    uuid "github.com/satori/go.uuid"
)

var enableMessageOrdering bool

func main() {
    flag.BoolVar(&enableMessageOrdering, "enableMessageOrdering", false, "Enable and use Pub/Sub message ordering")
    flag.Parse()

    client, err := pubsub.NewClient(context.Background(), "fleetsmith-dev")
    if err != nil {
        log.Fatalf("NewClient: %v", err)
    }

    topicID := "test-topic-" + uuid.NewV4().String()
    topic, err := client.CreateTopic(context.Background(), topicID)
    if err != nil {
        log.Fatalf("CreateTopic: %v", err)
    }
    topic.EnableMessageOrdering = enableMessageOrdering
    defer topic.Delete(context.Background())

    subID := "test-subscription-" + uuid.NewV4().String()
    sub, err := client.CreateSubscription(context.Background(), subID, pubsub.SubscriptionConfig{
        Topic:                 topic,
        EnableMessageOrdering: enableMessageOrdering,
    })
    if err != nil {
        log.Fatalf("CreateSubscription: %v", err)
    }
    defer sub.Delete(context.Background())

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    messageReceived := make(chan struct{})
    go sub.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
        log.Printf("Received message with ordering key %s: %s", msg.OrderingKey, msg.Data)
        msg.Ack()
        messageReceived <- struct{}{}
    })

    msg1, msg2 := &pubsub.Message{Data: []byte("Dang1!")}, &pubsub.Message{Data: []byte("Dang2!")}
    if enableMessageOrdering {
        msg1.OrderingKey, msg2.OrderingKey = "foobar", "foobar"
    }
    publishMessage(topic, msg1)
    publishMessage(topic, msg2)

    for i := 0; i < 2; i++ {
        select {
        case <-messageReceived:
        case <-time.After(10 * time.Second):
            log.Fatal("Expected to receive a message, but timed out after 10 seconds.")
        }
    }
}

func publishMessage(topic *pubsub.Topic, msg *pubsub.Message) {
    publishResult := topic.Publish(context.Background(), msg)
    messageID, err := publishResult.Get(context.Background())
    if err != nil {
        log.Fatalf("Get: %v", err)
    }
    log.Printf("Published message with ID %s", messageID)
}

当enableMessageOrdering标志设置为true时，我收到Dang1! 首先，其次是Dang2! ：

> go run main.go --enableMessageOrdering
2020/08/11 05:38:07 Published message with ID 1420685949616723
2020/08/11 05:38:08 Published message with ID 1420726763302425
2020/08/11 05:38:09 Received message with ordering key foobar: Dang1!
2020/08/11 05:38:11 Received message with ordering key foobar: Dang2!

而没有它，我会像以前一样以相反的顺序收到它们：

> go run main.go
2020/08/11 05:38:47 Published message with ID 1420687395091051
2020/08/11 05:38:47 Published message with ID 1420693737065665
2020/08/11 05:38:48 Received message with ordering key : Dang2!
2020/08/11 05:38:48 Received message with ordering key : Dang1!