GCP - Pub/Sub 发布速度变慢

Question

Few days ago (~2022. Feb. 09.), we detect a massive slowdown during publishing messages via ordered Google Cloud Pub/Sub . Before that day the publish time was around 1 sec, however that day this time increased above 15 sec. (Sometimes it was more than 30 sec.) That problem didn't come up constantly. The GCP status page did not show any related issue.

Our first thought was that the google lib ( nuget ) has some revealed bug, however when we tested the publish method via gcloud cli ( version ), the publish time was almost the same.

On this week the publishing works fine, and we can not reproduce it.

Does anybody else detect same issue? Is this normal with ordered topics? Should we prepare with 15 - 30 sec communication time via GCP - Pub/Sub?

This is how we publish our messages:

protected virtual async Task Publish<T>(T messageToSend, string topicName, string orderingKey = null)
{
    var hasOrdering = !string.IsNullOrEmpty(orderingKey);
    var message = new PubsubMessage
    {
        Data = ByteString.CopyFromUtf8(messageToSend.ToJson()),
        OrderingKey = orderingKey ?? string.Empty
    };

    var topic = new TopicName(projectId, topicName);
    var customSettings = hasOrdering 
        ? new PublisherClient.Settings
            {
                EnableMessageOrdering = true
            } 
        : null;

    PublisherClient publisher = await PublisherClient.CreateAsync(topic, settings: customSettings);

    await publisher.PublishAsync(message);

    logger.LogInformation($"Pubsub message sent for topic: {topicName}, OrderingKey: {orderingKey ?? "[empty]"}");

    await publisher.ShutdownAsync(TimeSpan.FromSeconds(15));
}

Do you have any related advice?

(We know that we dont use the latest version, however it seems that doesn't correlate with this issue. Furthermore as we wrote, we can not reproduce this issue anymore, so we can't test with new lib versions.)

Answer 1

There was a server-side issue between 2/7/2022 and 2/14/2022 that caused an unexpected slowdown in publishes for messages with ordering keys under specific publishing patterns. The scope was relatively limited, which is why it did not appear on the GCP status page. The issue was resolved by 2/14/2022 at 18:00:00 PST. This slow behavior is not normal for ordered topics and is not an issue anticipated to return.