Pulsar 消息总线：我们可以为每个主题配置消息保留吗？

Question

We have a usecase where our messages processor, processes messages with very complex calculations on messages from few topics. These calculations are not finishing within the rention period configured. Instead of increaseing message retention globally for all topics can we set this for the few topics where we need messages to be retained for more time? Message retention per topic is possible?

Answer 1

This is exactly the use case that namespaces were designed for. In Pulsar, a namespace is an administrative unit within a tenant that contains a subset of topics. The configuration policies set on a namespace apply to all the topics created in that namespace. The best way to address this issue is use the either the REST API or the pulsar-admin CLI tool to create a new namespace, eg

pulsar-admin namespaces create <current-tenant>/<a-new-namespace>

Once you have the new namespace, you can then configure the retention policy for the new namespace using the CLI tool, eg

$ pulsar-admin namespaces set-retention <current-tenant>/<a-new-namespace> \
  --size -1 \
  --time -1

Finally, you will want to re-create the topics in the new namespace, and change your code to use these new topics.

Answer 2

If you use a subscription on the topic, then you don't have to worry about message retention. Messages in a subscription backlog are not removed until they are acknowledged, regardless of the retention setting. Just use the Consumer interface in the client and acknowledge each message after the calculations are done.

For more details about retention and subscription backlogs, see this Understanding Pulsar Message TTL, Backlog, and Retention