Sleuth tracing is not working for transactional Kafka producers

Question

Currently, we are using transactional Kafka producers. What we have noticed is that the tracing aspect of Kafka is missing which means we don't get to see the instrumentation of Kafka producers thereby missing the b3 headers.

After going through the code, we found that the post processors are not invoked for transactional producers which means the TracingProducer is never created by the TraceProducerPostProcessor. Is there a reason for that? Also, what is the work around for enabling tracing for the transactional producers? It seems there is not a single place easily to create a tracing producer (DefaultKafkaProducerFactory #doCreateTxProducer is private)

Screen shot attached(DefaultKafkaProducerFactory class). In the screenshot you can see the post processors are invoked only for raw producer not for the case for transactional producer.

Your help will be much appreciated.

Thanks

DefaultKafkaProducerFactory#createRawProducer

Answer 1

??

createRawProducer() is called for both transactional and non-transactional producers:

Something else is going on.

EDIT

The problem is that sleuth replaces the producer with a different one, but factory discards that and uses the original.

https://github.com/spring-projects/spring-kafka/issues/1778

EDIT2

Actually, it's a good thing that we discard the tracing producer here; Sleuth also wraps the factory in a proxy and wraps the CloseSafeProducer in a TracingProducer ; but I see the same result with both transactional and non-transactional producers...

@SpringBootApplication
public class So67194702Application {

    public static void main(String[] args) {
        SpringApplication.run(So67194702Application.class, args);
    }

    @Bean
    public ApplicationRunner runner(ProducerFactory<String, String> pf) {
        return args -> {
            Producer<String, String> prod = pf.createProducer();
            prod.close();
        };
    }

}

Putting a breakpoint on the close() ...

Answer 2

Thanks Gary Russell for the very quick response. The createRawConsumer is effectivly called for both transactional and non transactional consumers.

Sleuth is using the TraceConsumerPostProcessor to wrap a Kafka consumer into a TracingConsumer. As the ProducerPostProcessor interface extends the Function interface, we may suppose the result of the function could/should be used but the createRawConsumer method of the DefaultKafkaProducerFactory is applying the post processors without using the return type. Causing the issue in this specific case.

So, couldn't we modify the implementation of the createRawConsumer to assign the result of the post processor. If not, wouldn't it be better to have post processors extending a Consumer instead of a Function?

Successful test made by overriding the createRawConsumer method as follow

    @Override
    protected Producer<K, V> createRawProducer(Map<String, Object> rawConfigs) {
        Producer<K, V> kafkaProducer = new KafkaProducer<>(rawConfigs, getKeySerializerSupplier().get(), getValueSerializerSupplier().get());
        for (ProducerPostProcessor<K, V> pp : getPostProcessors()) {
            kafkaProducer = pp.apply(kafkaProducer);
        }
        return kafkaProducer;
    }

Thank you for your help.