zookeeper-server-start.sh actually uses zookeeper jar from /kafka/libs not /zookeeper/lib. How to upgrade zookeeper?

Question

We have installed zookeeper 3.6.2 and Kafka 2.13-2.6.0 Recently I noticed that zookeeper-server-start.sh is actually using the zookeeper jar file from /kafka/libs/ which is zookeeper-3.5.8.jar.

1. How do I upgrade zookeeper to 3.6.2. Do I have to find a version of kafka that has it bundled in the tar?

2. Why do they ask you to download and install zookeeper if the jar is already bundled in kafka and that is the one that kafka uses?

Answer 1

You will need to separately install (and start) a Zookeeper cluster outside of the versions and scripts provided by Kafka if you want to upgrade it, but there are risks involved.

Kafka does come bundled with Zookeeper and is tested against specific versions, but it is not upgraded as frequently as Zookeeper project does releases. You should follow the Kafka upgrade notes before blindly upgrading dependency components; there are some notes around version 2.6 that mentions situations where Zookeeper upgrades can fail and what settings need to be added to fix it. And unless you can guarantee a Zookeeper server upgrade will be backwards compatible, Kafka still has a version-specific Zookeeper client for network communications.

Worth pointing out that Kafka KRaft (Zookeeper-less mode) is already in preview release phase with newest 3.x releases, and is scheduled to be closer to production-ready by the end of the year, so I would hold off on upgrading Zookeeper unless you actually need if for something else.