kafka 与 Apache Spark 的集成

Question

I'm learning apache spark integration with kafka so that my code could run automatically whenever new message arrives in Kafka's topic.

I've read official documentation as well

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html

But am still confused how its working. I've my word count written program in java spark and other side Kafka is running.

Is structured streaming a bridge between kafka and spark java code ? Does it keep listening to kafka and whenever message arrives, it pulls from kafka and pass it over to spark java code ..? Is it correct ..?

If not, can anyone please share with me, how it works in simple words ..? Any other reference will be appreciated.

How should I integrate my java spark code to Kafka so that it triggers automatically whenever new message arrives in kafka..?

Thanks

Answer 1

Spark delegates to the basic Kafka consumer APIs, which poll messages in batches as they arrive to the topic.

Structured Streaming and regular Spark Streaming work the same in this regard.

You may want to start with the Kafka basic consumer or Kafka Streams if you are interested in learning how Kafka record delivery works, as Spark might be overkill, depending on the task