Kafka：在不同版本之间复制主题吗？

Question

I've got two brokers. the first runs 0.9 and the second runs 0.10

Various workers and daemons consume and produce messages on both brokers.

For one application, I need messages from a 0.9 topic consumable from an application that's using KStreams and is connected to the 0.10 broker.

Is there a straightforward way of copying just the one topic from 0.9 to 0.10? Or using the 0.10 clients to connect to 0.9? I'd hate to have to resort to cramming both versions in the same jar. Just consuming the 0.9 broker with a 0.10 client doesn't seem to work.

Answer 1

In general, only Kafka broker are backward compatible (not Kafka client). Thus, a client can connect to newer brokers, but not to older once.

Because Kafka Streams library uses 0.10.x client, it only works with 0.10.x brokers.

Thus, upgrading your broker as described here: https://kafka.apache.org/documentation.html#upgrade should be the best way to go (this is safe, as brokers are backward compatible, thus not breaking any other applications using this broker with older clients).

As an alternative, you could also use Mirror Maker to replicate the topic from 0.9.x cluster to 0.10.x cluster.

Answer 2

my solution was to use jarjar to rewrite the 0.9 clients jar so that the classes/types don't conflict with the 0.10 client. it's dirty, but it works around the jvm's opinion on having two versions of the same library.