I am using Cassandra 2.2.8, JDK8, spark-cassandra-connector-java_2.10, spark-cassandra-connector_2.11-2.0.0-M3, cassandra-driver-core-3.1.0 and following Cassandra Spark Connector Example JavaDemo . This demo has to be fixed to compile with new 2.1 Connetcor API. I have fixed few things but this below one is stumping me: Compile Error at this line :
JavaPairRDD<Integer, BigDecimal> allSalesRDD = joinedRDD.flatMap(new PairFlatMapFunction<Tuple2<Integer, Tuple2<Sale, Product>>, Integer, BigDecimal>() {
@Override
public Iterable<Tuple2<Integer, BigDecimal>> call(Tuple2<Integer, Tuple2<Sale, Product>> input) throws Exception {
Error:
The method
flatMap(FlatMapFunction<Tuple2<Integer,Tuple2<SparkJavaDemo.Sale,SparkJavaDemo.Product>>,U>) in the type
AbstractJavaRDDLike<Tuple2<Integer,Tuple2<SparkJavaDemo.Sale,SparkJavaDemo.Product>>,JavaPairRDD<Integer,Tuple2<SparkJavaDemo.
Sale,SparkJavaDemo.Product>>> is not applicable for the arguments (new
PairFlatMapFunction<Tuple2<Integer,Tuple2<SparkJavaDemo.Sale,SparkJavaDemo.Product>>,Integer,BigDecimal>(){})
Thanks
You can use flatMapToPair
instead of flatMap
like below.
JavaPairRDD<Integer, BigDecimal> allSalesRDD = joinedRDD.flatMapToPair(new PairFlatMapFunction<Tuple2<Integer, Tuple2<Sale, Product>>, Integer, BigDecimal>() {
@Override
public Iterator<Tuple2<Integer, BigDecimal>> call(Tuple2<Integer, Tuple2<Sale, Product>> input) throws Exception {
Tuple2<Sale, Product> saleWithProduct = input._2();
List<Tuple2<Integer, BigDecimal>> allSales = new ArrayList<>(saleWithProduct._2().getParents().size() + 1);
allSales.add(new Tuple2<>(saleWithProduct._1().getProduct(), saleWithProduct._1().getPrice()));
for (Integer parentProduct : saleWithProduct._2().getParents()) {
allSales.add(new Tuple2<>(parentProduct, saleWithProduct._1().getPrice()));
}
return allSales.iterator();
}
});
I have created gist for updated code at https://gist.github.com/baghelamit/f2963d9e37acc55474559104f5f16cf1
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.