Kafka cluster restart via ansible

Question

What's the suggested way to restart all kafka server that are part of a cluster?

The nature of the question borns because we manage it via ansible , and we have a notify on changes regarding for example server.properties, so the restart will happen concurrently.

Usually when a broker disappear there is a re-balance of the data, so I'm wondering what happens if all brokers are rebooted at the same time.

Answer 1

Restarting an application in the sensible manner is tricky topic. To make it works you need to gather requirements:

• What is better - a single restart with possibility of no restart (at most once) or guaranteed restart with possibility of multiple restart (at least once)
• How to detect if the node is fine to restart?

Depending on the first question you may want to use file flags (in /tmp) to indicate that restart is pending (and use this flag as persistent storage for app restart if playbook failed in the middle).

For the second question the best way is to query something outside Ansible (fe some API server, monitoring server, etc) to see if restart is permitted.

The most safe and slow approach is to run restart playbook with serial: 1 and multiple checks before restarting the node.