来自 Pub/Sub 卡顿的 Beam(DataFlow) 流

Question

We have a Beam streaming job in DataFlow that reads from PubSub. During the peak hour amount of PubSub messages increases from 25k/s to 130k/s and stays at 120k/s for an hour. We can see a few short watermark spikes (at the beginning and a few after) in reading from Pub/Sub step(2-3minutes) and in the subscription backlog. Dataflow indicates Possible stuckness for all stages, but there is nothing in logs.

Our cluster is fixed-sized without autoscaling and I think we have enough workers(spikes go down fast, which means we can handle the peak hour load and even consume stuck backlog).

My only thought is that increased load in addition to future grouping aggregation can result in an inner shuffle in PubSub read. Maybe someone has any ideas on this?

Answer 1

You can use Ordering of messages to deliver messages in order, If messages have the same ordering key and are in the same region, you can enable message ordering and receive the messages in the order that the Pub/Sub service receives them.

Pub/Sub delivers each message at least once , so the Pub/Sub service might redeliver messages. When you receive messages in order and the Pub/Sub service redelivers a message with an ordering key, Pub/Sub maintains order by also redelivering the subsequent messages with the same ordering key. The Pub/Sub service redelivers these messages in the order that it originally received them.

Messages with the same ordering key are guaranteed to be delivered in order.

Messages with different ordering keys are not guaranteed to be delivered in order, independent of the publishing time.

In a rare case, the inability to acknowledge a message can hold up the delivery of messages for other ordering keys. This issue occurs when servers restart unexpectedly or there are changes in the set of servers used due to traffic changes. To preserve order across such events, all messages published to the old server must be acknowledged before messages from the new server are delivered, even if they are for different ordering keys.