Spark : Best way to Broadcast KafkaProducer to Spark streaming

Question

To broadcast KafkaProducer to spark executors I have created a wrapper like below :

public class KafkaSink implements Serializable {
    private static KafkaProducer<String, String> producer = null;

    public KafkaProducer<String, String> getInstance(final Properties properties) {
        if(producer == null) {
            producer = new KafkaProducer<>(properties);
        }
        return producer;
    }

    public void close() {
        producer.close();
    }
}

and using it like below

 JavaSparkContext jsc = new JavaSparkContext(sc);
 Broadcast<KafkaSink> kafkaSinkBroadcast = jsc.broadcast(new KafkaSink()));
 dataset.toJavaRDD().foreach(row -> kafkaSinkBroadcast.getValue().getInstance(kafkaProducerProps()).send(new ProducerRecord<String, String>(topic, row.mkString(", "))))

I just wanted to know whether its the right way to do it, or what is the best way to do it

Answer 1

I can really recommend this blog post . In short, you should create a serializable sink for each partition by passing a 'recipe' to create Kafka producer.