如何在一个微批量的 Spark 结构化流中设置批量大小

Question

I am reading streaming data from Kafka source, but all the data from kafka is read in a single micro batch.

spark.readStream.format("kafka").option("kafka.bootstrap.servers",bootstrap_servers).option("subscribe", topics).option("startingOffsets", "earliest").load()

Which parameter or option is used to set max batch size in one micro-batch of spark structured streaming?

Answer 1

use maxOffsetsPerTrigger to limit the no of messages.

as per spark doc "maxOffsetsPerTrigger - Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume."