Kafka producer - Sending a list of messages

Question

I need to send few batch of messages, and make sure that all messages in each batch arrives to the consumer together in the same batch.

For example, assume that I need to send 400 messages in 5 batches/groups, each group will contain 80 messages, and need to be consumed in the same batch in the consumer side.

I am using spark structured-streaming to consume the messages.

I read similar questions but i'm still confused about the right way of doing it.

Should the producer put all messages together (per batch) in a list, and send the list to kafka?

Is there any other better way?

Thanks

Answer 1

This can be achieved by creating a topic with 5 partitions, so that you can send each type of batch messages to each partition

ProducerRecord(java.lang.String topic, java.lang.Integer partition, K key, V value)
Creates a record to be sent to a specified topic and partition

And we can create 5 consumers and assign each consumer to each partition, but i'm not sure that each consumer poll() will pull the all the messages in that partition at a time or not

Manual Partition Assignment . here doc

For example: If the process is maintaining some kind of local state associated with that partition (like a local on-disk key-value store), then it should only get records for the partition it is maintaining on disk.
If the process itself is highly available and will be restarted if it fails (perhaps using a cluster management framework like YARN, Mesos, or AWS facilities, or as part of a stream processing framework). In this case there is no need for Kafka to detect the failure and reassign the partition since the consuming process will be restarted on another machine.
To use this mode, instead of subscribing to the topic using subscribe, you just call assign(Collection) with the full list of partitions that you want to consume.

 String topic = "foo";
 TopicPartition partition0 = new TopicPartition(topic, 0);
 TopicPartition partition1 = new TopicPartition(topic, 1);
 consumer.assign(Arrays.asList(partition0, partition1));