如何使用Scala 2.11将Kafka作为Apache Spark的流运行？

Question

I haven't been able to find any any build of Spark Streaming integration for Kafka for Scala 2.11. There is a one available here http://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka_2.10 but it is for 2.10

Can anyone point me to a 2.11 build?

Answer 1

Specific to your question:

It's not feasible to run Spark Kafka against Scala 2.11 by now ( Spark-1.3 )

General method to build from source code:

If no pre-build version available, you could build spark yourself and fulfil your needs by specifying some build parameters.

The detailed build procedure could be find: Building Spark

In short, It would only take two steps if build against scala-2.10:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

mvn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

You should specify profiles or properties that fits your situation in the second command

Note the part Building Spark states on Building for Scala 2.11 :

To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 property:

dev/change-version-to-2.11.sh
mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package

Scala 2.11 support in Spark does not support a few features due to dependencies which are themselves not Scala 2.11 ready. Specifically, Spark's external Kafka library and JDBC component are not yet supported in Scala 2.11 builds.