How to determine topic for a Kafka message

Question

I have code like below to read from Kafka and save to Elasticsearch, I am using spark streaming:

JavaDStream<String> liness = messages.map(new Function<ConsumerRecord<String, String>, String>() {
        @Override
        public String call(ConsumerRecord<String, String> kafkaRecord) throws Exception {
            System.out.println(kafkaRecord.topic() + kafkaRecord.value());
            return kafkaRecord.value();
        }
    });

        JavaEsSparkStreaming.saveJsonToEs(lines, "events/redict");
        jssc.start();

My problem is that I need to make "events/redict" dynamic depending on the name of the topic, I also am able to fetch the topic name at messages.map call however how do I add the dynamism here?

Answer 1

You would add "dynamism" here by either making one stream per topic, or filtering your messages stream into events and redict RDDs.

Individually, you save those two to the proper ES indicies.