多个 Kafka 分区到 Akka Streams

Question

Hi I am working with Kafka and Akka Streams . In Kafka for a topic MyTestTopic I have 3 partitions and data is being pushed into the topic at high concurrency roughly 1000 QPS and it'll only go higher than that.

Below is my code for Akka Stream Kafka Consumer:

final ConsumerSettings<String, byte[]> consumerSettings =
        ConsumerSettings.create(kafkaConfig, new StringDeserializer(), new ByteArrayDeserializer())
                .withBootstrapServers("127.0.0.1:9092")
                .withGroupId("TestConsumerGroup")
                .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
                .withProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false")
                .withProperty(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, String.valueOf(timeout));

ActorMaterializer materializer = ActorMaterializer.create(system);

RestartSource.onFailuresWithBackoff(
        java.time.Duration.ofSeconds(3),
        java.time.Duration.ofSeconds(3000),
        0.2,
        () -> Consumer.atMostOnceSource(consumerSettings, Subscriptions.topics("MyTestTopic"))
                .mapAsyncUnordered(10,
                        record -> ask(rootHandler, new StreamData(record), Duration.ofSeconds(timeout))))
        .to(Sink.foreach(App::sinkParser))
        .run(materializer);

My Question:

• How can I define multiple Akka Stream consumers to listen to different Kafka partitions since multiple partitions leading to a single instance of Akka Steam seems like a bottle-neck .
• Is Akka Clustering the answer to this? Keeping 2 seed nodes on static servers and multiple akka stream consumers on auto scale in a cloud based environment.

I can't seem to figure it out, I need help thanks

Answer 1

There are a couple of ways to approach this, depending on particulars you haven't elaborated on:

If you're reasonably sure that one node can handle processing all the messages, you can set up multiple streams up to 1 stream per partition.

An evolution of this would be to use a CommittablePartitionedSource so that you dynamically create as many streams as there are partitions. Note that you'll need to manually commit offsets (eg using Committer.sink ).

You can have one stream per instance and deploy up to as many instances as you have partitions; with the same consumer group, the instances will coordinate among themselves the partition assignments. When deploying multiple instances, you may or may not need Akka Cluster, depending on the nature of what the actor you're ask ing is doing.

If no state is being maintained in the actor per message (note that this would encompass the actor doing a read-modify-write on an external datastore: if you can ensure that the messages affecting a given row are in the same Kafka partition, you might even be able to do without ACID in that external datastore) you likely don't need Akka clustering.

If the actors are themselves stateful (eg they're shadowing some IoT device), then you almost certainly want the combination of Akka Cluster, Akka Cluster Sharding, and Akka Persistence. Going all the way with this does have some advantages over actors doing read-modify-write on an external datastore (eg most of the reads can be eliminated in favor of tracking state in the actor and embracing event sourcing).