For microservice, how to add nodes for Kafka and tb-rule-engine at runtime, will it impact availability?

Question

like how re-hash after adding nodes. Will there be some period of time of unavailability?

There are no official documents for this answer, so I wonder if Thingsboard supports smooth dynamic expansion.

Answer 1

ThingsBoard Rule Engine uses Zookeeper to detect the addition/deletion of sibling nodes. The data stream of Rule Engine messages is stored in Kafka topic with a configurable amount of partitions. When you add the node, the rule engine PartitionService.recalculatePartitions method is executed on each node to get the list of partitions that this node is responsible for. So, old Rule Engine nodes will stop consuming a certain partition and the new rule engine node will start consuming those partitions. During a short period of time, you may notice a slight degradation in processing speed (due to the warm-up of the new node) but this should not affect the data processing logic itself.

As a side effect, some of the messages may be processed twice in this case. For example, rule engine A polled 1000 messages from the queue but has not committed the offset yet. The messages are now traveling through the rule chains. Then the 2nd node started and starts reading the same partition. It will start processing the uncommitted messages again.

If you must avoid duplicates in processing - configure your queue to process messages one-by-one (Sequentially, with the size of the pack equal to 1). Although this will cause performance degradation.