使用Kafka消耗spark RDD

Question

Are there any samples in JAVA to consume spark RDD in JAVA using Kafka, If there is one please let me know

I am trying out a few things but i am unsure as to how to send an RDD in a producer and consume it in Kafka consumer.

Thanks.

Answer 1

For writing your RDD to kafka you can use cloudera's API. Here is the link .

Answer 2

Code will be like this:

import org.apache.spark.streaming.kafka.*;

 JavaPairReceiverInputDStream<String, String> directKafkaStream = 
     KafkaUtils.createDirectStream(streamingContext,
         [key class], [value class], [key decoder class], [value decoder class],
         [map of Kafka parameters], [set of topics to consume]);

Have a look at kafka-integration and code example