Kafka 流联合分区与交互式查询

Question

I have the following topology:

topology.addSource(WS_CONNECTION_SOURCE, new StringDeserializer(), new WebSocketConnectionEventDeserializer()
            , utilService.getTopicByType(TopicType.CONNECTION_EVENTS_TOPIC))
            .addProcessor(SESSION_PROCESSOR, WSUserSessionProcessor::new, WS_CONNECTION_SOURCE)
            .addStateStore(sessionStoreBuilder, SESSION_PROCESSOR)
            .addSink(WS_STATUS_SINK, utilService.getTopicByType(TopicType.ONLINE_STATUS_TOPIC),
                    stringSerializer, stringSerializer
                    , SESSION_PROCESSOR)

            //WS session routing
            .addSource(WS_NOTIFICATIONS_SOURCE, new StringDeserializer(), new StringDeserializer(),
                    utilService.getTopicByType(TopicType.NOTIFICATION_TOPIC))
            .addProcessor(WS_NOTIFICATIONS_ROUTE_PROCESSOR, SessionRoutingEventGenerator::new,
                    WS_NOTIFICATIONS_SOURCE)
            .addSink(WS_NOTIFICATIONS_DELIVERY_SINK, new NodeTopicNameExtractor(), WS_NOTIFICATIONS_ROUTE_PROCESSOR)
            .addStateStore(userConnectedNodesStoreBuilder, WS_NOTIFICATIONS_ROUTE_PROCESSOR, SESSION_PROCESSOR);  

As you can see there are 2 source topics. State store is built from the first topic and the second flow reads the state store. When I start the topology, I see those stream threads are assigned the same partitions (co-partitioning) of both source topics. I assume this is because the state store is accessed by the second topic flow.

This is functionally working fine. But there is a performance problem. When there is a surge in the volume of input data to the first source topic, which updates state-store, second topic processing is delayed.

For me, the second topic should be processed as fast as possible. Delay in processing the first topic is fine.

I am thinking of the following strategy:

Current configuration:
     WS_CONNECTION_SOURCE - 30 partitions
     WS_NOTIFICATIONS_SOURCE - 30 partitions
     streamThreads: 10
     appInstances: 3 

New configuration:
    WS_CONNECTION_SOURCE - 15 partitions
    WS_NOTIFICATIONS_SOURCE - 30 partitions
    streamThreads: 10
    appInstances: 3
    Since there is no co-partitioning, tasks has to use interactive query to access store

The idea is out of 10 threads, 5 threads will only process the second topic which can alleviate the current problem when there is a surge in the first topic.

Here are my questions:

1. Is this strategy correct? To avoid co-partitioning and use interactive query
2. Is there a chance that Kafka will assign 10 partitions of WS_CONNECTION_SOURCE 
   to one instance since there are 10 threads and one instance won't get any?
3. Is there any better approach to solve the performance problem?

Answer 1

State store and Interactive Query are Kafka Streams abstraction. To use Interactive Query you have to define state store (using Kafka Streams API) and that enforce you to have same number of partitions, for inputs topics. I think your solution will not work. Interactive query are for exposing ability to query state store outside the Kafka Streams (not for access within Processor API)

Maybe you can review your SESSION_PROCESSOR source code and extract more work to Process from the other topology and publish result to intermediate topic and then based on that build that state store.

Additionally:

Currently Kafka Streams doesn't support prioritization for input topics. There is KIP about priorities for Source topic: KIP-349 . Unfortunately linked Jira ticket was closed as Won't FIX ( https://issues.apache.org/jira/browse/KAFKA-6690 )