如何处理kafka生产者异常

Question

I am trying to understand how spring boot KafkaTemplate works with async producer and handle exceptions. I want to handle all kinds of errors including.network errors. Tried with retry configs but its retrying more than the number I provided to

@Service
public class UserInfoService {
     private static final Logger LOGGER = LoggerFactory.getLogger(UserInfoService.class);

     @Autowired
     private KafkaTemplate kafkaTemplate;

     public void sendUserInfo(UserInfo data) {
         final ProducerRecord<String, UserInfo> record = new ProducerRecord<>("usr-test-data", "test-app", data);
         
         try {
             ListenableFuture<SendResult<String, UserInfo>> future = kafkaTemplate.send(record);
             future.addCallback(new ListenableFutureCallback<SendResult<String, UserInfo>>() {
                 @Override
                 public void onFailure(Throwable ex) {
                     handleFailure(ex);
                 }

                 @Override
                 public void onSuccess(SendResult<String, UserInfo> result) {
                     handleSuccess(result);
                 }
             });
         } catch (Exception e) {
             throw new RuntimeException(e);
         }
         
     }

     private void handleSuccess(SendResult<String, UserInfo> result) {
         LOGGER.info("Message sent successfully with offset: {}", result.getRecordMetadata().offset());
     }
     private void handleFailure(Throwable ex) {
         LOGGER.info("Unable to send message- Error: {}",  ex.getMessage());
     }
 }

Tried to limit number of retries with configProps.put(ProducerConfig.RETRIES_CONFIG, "3"); hoping this will eventually throw exception and I can catch. But it still tries more than 3 times and seems not working. Here is my complete config class:

@Configuration
public class KafkaProducerConfig {
     @Value(value = "${spring.kafka.bootstrap-servers}")
     private String bootstrapAddress;

     @Bean
     public ProducerFactory<String, String> producerFactory() {
         Map<String, Object> configProps = new HashMap<>();
         configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
         configProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_DOC, "true");
         configProps.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, CountingProducerInterceptor.class.getName());
         configProps.put(ProducerConfig.ACKS_CONFIG, "all");
         configProps.put(ProducerConfig.RETRIES_CONFIG, "3");
         configProps.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 10000);
         configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
         configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
         return new DefaultKafkaProducerFactory<>(configProps);
     }

     @Bean
     public KafkaTemplate<String, String> kafkaTemplate() {
         return new KafkaTemplate<>(producerFactory());
     }
 }

Would like to know what I can catch in future.OnFailure and in the parent try catch block.

Answer 1

A Future does not complete within the lifecycle of a try-catch. You need to throw the exception within the body of the onFailure.

In my experience, Kafka.network errors cannot easily be caught