添加空处理器时，Kafka Streams的处理速度变慢

Question

Consider this Kafka Streams driver

public class TestDriver {

    private static final String SOURCE = "SOURCE";

    public static void main(String[] args) throws Exception {

        ProtoDeserializer<Message> protoDeserializer = new ProtoDeserializer<>(Message.parser());
        ProtoSerializer<Message> protoSerializer = new ProtoSerializer<>();

        StringDeserializer stringDerializer = new StringDeserializer();
        StringSerializer stringSerializer = new StringSerializer();

        Topology topologyBuilder = new Topology();
        topologyBuilder.addSource(SOURCE, stringDerializer, protoDeserializer, "input-messages")

            .addProcessor(DummyProcessor.NAME, DummyProcessor::new, SOURCE)

            .addSink("MAIN", "output-messages", stringSerializer, protoSerializer, DummyProcessor.NAME)
        ;

        KafkaStreams streams = new KafkaStreams(topologyBuilder, getConfig());
        streams.cleanUp();
        streams.start();

        System.out.println(streams.toString());

        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

    }

    private static Properties getConfig() {
        Properties config = new Properties();
        config.put(StreamsConfig.CLIENT_ID_CONFIG, "test.stream-processor");
        config.put(StreamsConfig.APPLICATION_ID_CONFIG, "test.stream-processor");
        config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "broker-1:9092,broker-2:9092,broker-3:9092");
        config.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3);
        config.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 10);
        config.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, WallclockTimestampExtractor.class);
        return config;
    }
}

The question is that when no processor is added to the topology (no .addProcessor() is included), the processing speed from source to sink is fine (means that I currently produce 25k messages/s and it has no problem to catch up).

However, when DummyProcessor is added, it suddenly processes 3k messages/s max (600k bytes).

DummyProcessor does basically nothing:

public class DummyProcessor extends AbstractProcessor<String, Message> {

    public static final String NAME = "DUMMY_PROCESSOR";

    public void process(String key, Message originalMessage) {
        context().forward(key, originalMessage);
        context().commit();
    }
}

Is adding single "empty" processor such overhead for Streams performance? What is the cause of it? Is Kafka Streams so smart that when there's not processor it doesn't perform protobuf serde and only forwards data received? Anyway to speed it up?

With such speed I'd need like x thousands more cpu threads available to be able to process all my data as 25k messages/s is 1 % of what I have. That sounds a lot.

Answer 1

Issue is caused because of requesting commit to often.

You don't need to call ProcessorContext:commit() at all. Kafka Streams based on commit.interval.ms property performs commit (be default: 30000 ms ). If exactly once semantic is set its different value. You can details in https://kafka.apache.org/documentation/#streamsconfigs .

If in some use case you need commit more frequently you can call ProcessorContext:commit() . But you have to remember, that commit is not made Immediately (directly). It sets only flag to commit as soon it will be possible.