Kafka producer and broker compression

Question

If Kafka producer compression is set (eg to gzip), and the broker configuration is also set to the same codec, will the broker re-compress any messages from the producer, or recognise that its the same codec and skip and broker-side re-compression?

I'm aware that the broker can be configured to inherit broker codec via the 'producer' setting. However, for our scenario we may have producers (out of our control) who may not set any compression, so we'd like to configure the broker to have default compression enabled, but for those producers that are in our control we'd prefer to use producer compression to save on network bandwidth but also the reduce load on the broker.

Answer 1

Setting topic compression to producer is equivalent to setting it to the same value you use in your producers. Thus to achieve what you need, you just need to set topic compression to the same algo you use in your producers. The external producers that use the same compression algorithm will work the same as your internal producers, and the rest will trigger a potential decompression/recompression.

This article sums it up nicely: https://newbedev.com/if-i-set-compression-type-at-topic-level-and-producer-level-which-takes-precedence