Java中带有Spark 2.1.0的FlatMap

Question

我正在尝试使用Java 8的spark 2.1.0与flatMap配合使用

2.2.0文档显示了此示例

JavaDStream<String> words = lines.flatMap(x -> Arrays.asList(x.split(" ")).iterator());

当我从2.1.0开始尝试时，得到以下内容

Error:(31, 25) java: method flatMap in class org.apache.spark.rdd.RDD<T> cannot be applied to given types;
required: scala.Function1<java.lang.String,scala.collection.TraversableOnce<U>>,scala.reflect.ClassTag<U>
found: (x)->Array[...]tor()
reason: cannot infer type-variable(s) U
(actual and formal argument lists differ in length)

给定这些版本，flatMap的正确方法是什么？

Answer 1

下面的代码适用于Spark 2.1.0。

JavaDStream<String> lines = messages.map(tuple -> tuple._2());
JavaDStream<String> words = lines.flatMap(x -> Arrays.asList(SPACE.split(x)).iterator());
JavaPairDStream<String, Integer> wordCounts = words.mapToPair(s -> new Tuple2<>(s, 1))
    .reduceByKey((i1, i2) -> i1 + i2);

请检查pom.xml以获取火花依赖版本。 如果要引用Spark 2.1.0版本的示例，请转到https://github.com/apache/spark/tree/branch-2.1/examples/src/main/java/org/apache/spark/examples/streaming