How to receive a Java Object from Kafka in Spark Streaming Application

Question

I have many objects of a class Say Test which I want to write to Kafka and process them using spark streaming App. I want to use the Kryo Serialization.

My application is in Java

JavaDStream<Test> testData = KafkaUtils
                .createDirectStream(context , keyClass,valueClass ,keyDecoderClass ,valueDecoderClass , props,topics);

My question is what should I put for keyClass,valueClass ,keyDecoderClass ,valueDecoderClass ?

Answer 1

Say if your topic is "String " and value is "Test" then first you would need to create TestEncoder and TestDecoder classes by implementing kafka.serializer.Encoder and kafka.serializer.Decoder . Now in your createDirectStream method you can have

JavaPairInputDStream<String, Test> testData = KafkaUtils
            .createDirectStream(context, String.class,Test.class ,StringDecoder.class,TestDecoder.class,props,topics);

You can refer KafkaKryoEncoder at https://www.tomsdev.com/blog/2015/storm-kafka-complex-types/

In your Kafka producer you would need to register your custom Encoder class like

Properties properties = new Properties();
properties.put("metadata.broker.list", brokerList);
properties.put("serializer.class", "com.my.TestEncoder");
Producer<String, Test> producer = new Producer<String, Test>(new ProducerConfig(properties));
Test test = new Test();
KeyedMessage<String, Test> data = new KeyedMessage<String, Test>("myTopic", test);
producer.send(data);