Scala外壳无法识别火花

Question

I am trying to subscribe to a topic in kafka using scala shell and below is the command that i am executing for the same:

scala>val topic1 = spark.read.format("kafka").option("subscribe","topic1").option("kafka.bootstrap.servers", "ip:port,ip2:port,ip3:port").load

whenever i execute the command i end up in error as below:

:25: error: not found: value spark Is there any other way i can subscribe to a topic in kafka via scala shell. I am usingSpark - 1.6.3 Scala - 2.10.5 version.

Answer 1

Spark is not available in standard scala lib, but you can load external libs like spark by adding jar files to classpath fe:

scala -cp SomeLib.jar

Or with :require

scala> :require SomeLib.jar

And then you will need to import proper package. But question is why? If you want to process that messages somehow then it will be much easier to use some simple project with build tool like SBT. And if you want just to consume messages fe. for some testing / preview and you don't want to build custom app for that, then there is a kafka-console-consumer in Kafka: https://kafka.apache.org/quickstart#quickstart_consume

Answer 2

You can use spark-shell which looks exactly like Scala shell but with all necessary libraries available and spark context created. spark-shell is included in a standard Spark installation.

See eg https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-shell.html for details.