Kafka Stream - 只有一个实例从分区读取

Question

I'm trying to start 2 instances of my application to read messages from the same topic with 30 partitions, but only the first instance I start reads the messages (from all 30 partitions).

Both instances have the same configuration:

private Properties streamConfig() {
    streamConfig = new Properties();
    streamConfig.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    streamConfig.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, "myApp");
    streamConfig.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 30);
    streamConfig.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3);
    streamConfig.setProperty(StreamsConfig.producerPrefix(ProducerConfig.ACKS_CONFIG), "all");
    return streamConfig;
}

What am I doing wrong?

Thanks.

Answer 1

Try to set lower NUM_STREAM_THREADS_CONFIG. Each thread is actually a separate consumer. So what might be happening is first instance getting all partitions assigned to it. When second instance starts rebalance happening but it not necessarily splits partitions evenly between instances (or even assigns something to second instance). In any case in setup you described 30 threads will be running idle when both instances are up.