水平縮放 spring-kafka 消費者應用程序

Question

我想知道什么是配置與水平縮放實例的最大數量相關的分區數量的好方法。

假設我有一個包含6 個分區的主題。

我有一個應用程序使用ConcurrentKafkaListenerContainerFactory和setConcurrency為6 。 這意味着我將有6 個KafkaMessageListenerContainer ，每個使用一個線程並使用來自我所有分區的均勻分布的消息。

如果以上是正確的，那么我想知道如果我通過添加另一個實例水平縮放應用程序會發生什么？ 如果新實例具有相同的 6 並發配置，當然還有相同的消費者組，我相信第二個實例將不會使用任何消息。 因為不會發生重新平衡，因為每個現有的消費者都會分配一個分區。

但是，如果我們 go 回到第一個示例並有6 個分區，其中一個實例的並發為 3，那么每個消費者線程/ KafkaMessageListenerContainer將分配2 個分區。 如果我們擴展這個應用程序（相同的消費者組 ID 和 3 的並發），我相信會發生重新平衡，並且兩個實例將分別從 3 個分區中消費。

這些假設是否正確？如果不正確，您應該如何處理這種情況？

Answer 1

一般來說，您的假設對於默認行為是正確的，它基於：

/**
 * <p>The range assignor works on a per-topic basis. For each topic, we lay out the available partitions in numeric order
 * and the consumers in lexicographic order. We then divide the number of partitions by the total number of
 * consumers to determine the number of partitions to assign to each consumer. If it does not evenly
 * divide, then the first few consumers will have one extra partition.
 *
 * <p>For example, suppose there are two consumers <code>C0</code> and <code>C1</code>, two topics <code>t0</code> and
 * <code>t1</code>, and each topic has 3 partitions, resulting in partitions <code>t0p0</code>, <code>t0p1</code>,
 * <code>t0p2</code>, <code>t1p0</code>, <code>t1p1</code>, and <code>t1p2</code>.
 *
 * <p>The assignment will be:
 * <ul>
 * <li><code>C0: [t0p0, t0p1, t1p0, t1p1]</code></li>
 * <li><code>C1: [t0p2, t1p2]</code></li>
 * </ul>
 *
 * Since the introduction of static membership, we could leverage <code>group.instance.id</code> to make the assignment behavior more sticky.
 * For the above example, after one rolling bounce, group coordinator will attempt to assign new <code>member.id</code> towards consumers,
 * for example <code>C0</code> -&gt; <code>C3</code> <code>C1</code> -&gt; <code>C2</code>.
 *
 * <p>The assignment could be completely shuffled to:
 * <ul>
 * <li><code>C3 (was C0): [t0p2, t1p2] (before was [t0p0, t0p1, t1p0, t1p1])</code>
 * <li><code>C2 (was C1): [t0p0, t0p1, t1p0, t1p1] (before was [t0p2, t1p2])</code>
 * </ul>
 *
 * The assignment change was caused by the change of <code>member.id</code> relative order, and
 * can be avoided by setting the group.instance.id.
 * Consumers will have individual instance ids <code>I1</code>, <code>I2</code>. As long as
 * 1. Number of members remain the same across generation
 * 2. Static members' identities persist across generation
 * 3. Subscription pattern doesn't change for any member
 *
 * <p>The assignment will always be:
 * <ul>
 * <li><code>I0: [t0p0, t0p1, t1p0, t1p1]</code>
 * <li><code>I1: [t0p2, t1p2]</code>
 * </ul>
 */
public class RangeAssignor extends AbstractPartitionAssignor {

但是，您可以通過partition.assignment.strategy消費者屬性插入任何ConsumerPartitionAssignor ： https://kafka.apache.org/documentation/#consumerconfigs_partition.assignment.strategy

另請參閱ConsumerPartitionAssignor JavaDocs 了解更多信息及其實現，以便為您的用例做出選擇。