使用 Scala/Spark 编程计算 kafka 主题中的消息数量

Question

The requirement is to count the number of messages in a Kafka topic using Scala or Spark programming. I am new to both the programming so i am not sure how can it be done. Can anybody help me with the code or guide me how can it be achieved.

Answer 1

The methods for a given org.apache.kafka.clients.consumer.KafkaConsumer are: beginningOffsets(Collection<TopicPartition>) and endOffsets(Collection<TopicPartition>) .

You need to remember that messages can be removed by kafka broker, that's why there is a need for beginning ones.