如何在PySpark中创建kafka流？

Question

I'm trying to create a kafka stream and then do some transformations on that but It seems the stream that I create is null. I load a text file into producer then consume it through consumer and It works fine but it doesn't create the kafka stream. The input text file looks like this with 36000 entries:

10.000000
26.000000
-8.000000
-28.000000
...

And my python code is:

sc = SparkContext(appName="STALTA")
ssc = StreamingContext(sc, 2)
broker, topic = sys.argv[1:]
kvs = KafkaUtils.createStream(ssc, broker, "raw-event-streaming-consumer",{topic:1})
rdd = kvs.flatMap(lambda line: line.strip().split("\n")).map(lambda strelem: float(strelem))
print("****** ", rdd.count())
ssc.start()
ssc.awaitTermination()

rdd.count() should print 36000 but it's empty.

The command that I run my script with is the following:

bin/spark-submit --jars jars/spark-streaming-kafka-0-8-assembly_2.11-2.3.1.jar examples/src/main/python/streaming/sparkkafka.py localhost:2181 consumer6

I tried localhost:9092 also but it didn't work.

Do you know what I'm doing wrong? Thank you.

Answer 1

I think you should change the broker to the zookeeper IP, usually it's localhost:2181 because it required the zookeeper IP not the kafka IP.

For the question to print the DStream , you could directly print with kvs.pprint() usually it will print like a tupple of rdd. Don't use the print() its not recognized by the sparkstreaming as output command so you will get those error.

Error : java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute

Hope this will help you.