spring-kafka: DeadLetterPublishingRecoverer vs RetryTopicConfiguration

Question

How are DefaultErrorHandler and RetryTopicConfiguration/Builder related? Can they work together or does one take precedence over the other?

I am trying to setup my consumption such that

1. Retry a certain number of times ( FixedBackOff(attempts = 3, interval = 5s) )
2. Use a single topic for retry ( RetryTopicConfiguration.useSingleTopicForFixedDelays works)
3. Customize the name of the retry topic (say main-topic-name.retry ). Have not found any class/method that lets me do that.
4. If the retries are exhausted, then post to a dead letter with a customized name ( main-topic-name.deadLetter ). DeadLetterPublishingRecoverer second parameter BiFunction should work but looks like the DL topic has to created beforehand. Not entirely sure.

Observations

• If I use DeadLetterPublishingRecoverer without retry related property configuration, I am able to retry with the hard-coded FixedBackOff ( return new DefaultErrorHandler(recoverer, new FixedBackOff(5000, 3)); )
• If I use DeadLetterPublishingRecoverer with retry related property configuration, the above hard-coded backoff is overridden. I am puzzled how the override happens even though I am supplying a configured error handler instance.
• By using DeadLetterPublishingRecoverer only, I don't get a separate retry topic.
• By using RetryTopicConfiguration (with or without recoverer), I get a retry topic, not with the name I want but default one. However, the dead letter topic used is not mine, the one supplied in DeadLetterPublishingRecoverer but the default one ( ....-dlt )

I am hoping people would have come across this and have a found a solution to the way I intend to setup my consumption (numbered list above).

Help appreciated

Answer 1

Your question seems to be more about topic naming than about DeadLetterPubishingRecoverer .

If you just need to change the topics' suffix, such as main-topic-name.retry and main-topic-name.deadLetter , you can use methods in the RetryTopicConfigurationBuilder , such as:

@Bean
public RetryTopicConfiguration myRetryTopic(KafkaTemplate<String, MyOtherPojo> template) {
    return RetryTopicConfigurationBuilder
            .newInstance()
            .useSingleTopicForFixedDelays()
            .retryTopicSuffix(".retry")
            .dltTopicSuffix(".deadletter")
            .create(template);
}

If you need more customization than that, you can also provide a RetryTopicNamesProviderFactory .

Refer to the feature's topic naming documentation - versions 2.8.x and 2.9.x for more details.

About the interaction with DefaultErrorHandler and DeadLetterPubishingRecoverer , the way the feature works is the framework will setup its own DEH and DLPR to handle record forwarding to the retry topics and dlt.

You can customize these components if necessary - for 2.9.x see this section of the documentation on how to configure it.