spark-如何在另一个RDD转换中查找（Java）PairRDD的键和值

Question

I have a PairRDD rdd1 with Integer keys and Integer[] values.

I also have another PairRDD rdd2 with Integer keys and Double values.

Each Integer in the key AND the value of rdd1 also exists in rdd2 as a key.

I want for each pair (x, [y1,y2,...,yn]) in rdd1 to get the double value of the x and all the double values of each Integer y1 , y2 , ..., yn .

I tried collecting rdd2 as a Map<Integer,Double> ( map2 ), but it does not fit in memory and I get OOM errors. I also tried joining the rdds, but I could not figure out how to join both the keys and the values. Using rdd2 's lookup() method inside rdd1 is not allowed.

The pseudocode of what I want is the following:

map each (int x, int[] y) in rdd1 to:
      (x, map2.get(x) + sum(map2.get(yi)))

for each yi in y .

I use Java, but I guess the same problem holds in both Java and Scala.

Answer 1

Depending on what you want to do with missing matches (cases where there's an index in rdd1 and no corresponding index in rdd2 ), the query looks something like the following.

rdd1.
    // ( x, [ y1, ..., yn ] ) -> ( x, x ), ( y1, x ), ..., ( yn, x )
    flatMap { case ( x, ys ) => ( x :: ys ).map( ( _, x ) ) }.
    // ( xory, x ) -> ( xory, ( x, rdd2.lookup( xory ) ) )
    leftOuterJoin( rdd2 ).
    // ( xory, ( x, rdd2.lookup( xory ) ) ) -> ( x, rdd2.lookup( xory ) )
    map( _._2 ).
    // ( x, rdd2.lookup( x ) ), ... -> ( x, rdd2.lookup( x ) + sum_i( rdd2.lookup( y_i ) )
    reduceByKey{ case ( dopt1, dopt2 ) => ( dopt1 ++ dopt2 ).reduceOption( _ + _ ) }.
    // unwrap the option types
    mapValues( _.getOrElse( 0.0 ) )

Answer 2

HashMap<Integer, List<Integer>> map = new HashMap<>();
    map.put(1,asList(2,3));
    map.put(3,asList(4,5));

    System.out.println(
            map.entrySet().stream()
                    .flatMap(kv -> 
                            Stream.concat(
                                    Stream.of((double)kv.getKey()),
                                    kv.getValue().stream().mapToDouble( x -> Double.valueOf((double)x) ).boxed())
                    )
                    .collect(Collectors.toList())
            );

How about this? ... should give you all (keys and values) in one RDD which you can use as keys in your second RDD. You can of course change the type.