使用 Apache Beam 从 PubSubIO 获取 Pub/Sub 消息的 messageId 字段

Question

Is it possible the get somehow the messageId field of a Pub/Sub message in a DoFn after using the PubSubIO Beam source to read the messages?

I need the default id which was assigned by the Pub/Sub service. I want to log it for debugging purposes.

Using a custom attribute for the unique id and the withIdAttribute() method is not possible for me, because I have no influence on the publisher in this case.

I use the 2.2.0 version of the Dataflow Java SDK.

Answer 1

Support for reading the Pubsub message id was added starting with Beam v2.16.0. To turn it on, replace .readMessages() with .readMessagesWithMessageId() in your pipeline setup then it is as easy as message.getMessageId() after that change.

Answer 2

For debugging purposes you can use the seek option .

It creates a snapshot of the messages which you can replay when needed.