Spark KafkaUtils CreateRDD apply filter on key
Question
I have a huge Kafka topic that contains several key,value messages. I want just to process data with a given key in an RDD, without having to download the whole topic. Data is also interleaved, so I cannot even rely on the offset in the topic Any suggestion on how to perform this?
1 answers
solution1
1 2021-05-30 11:57:32
Not possible. Need to filter them (all).
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Creating a JavaPairRDD using KafkaUtils.createRDD (spark and kafka)
Spark Streaming doesn't contain KafkaUtils
Unable to create Java input dstreams using KafkaUtils :Spark 1.6.0
how to apply dictionary key to value project to a column in dataset in spark?
how to apply filter on composite-id key property in hibernate
Apply dataset as Broadcast in Spark
filter KeyValueGrouped Dataset in spark
Optionally apply filter on stream
Apply filter for GHE issues
Apply filter on Page objects
Related Tags