如何有效地加入两个表 - 大表和小表？

Question

I have 2 data sets.One is big and one more small data set.I was processing the data in map reduce by putting the small data sets in distributed cache and getting it in mapper and performing join with some more operations.

I want this to move into spark java programming.But am getting only a map function where can i transform my rdd and for distributed cache am getting to broadcast the rdd ,but i am not getting how to pass the broadcast variable to the map function.

 JavaPairRDD<String, String> logData = sc.wholeTextFiles(args[0]);
     logData.map(new Transformation());
     String [] vals={"val,hel","hi,by"};
     JavaRDD<String>javaRDD=sc.parallelize(Arrays.asList(vals));
     Broadcast<String> broadcastVar=sc.broadcast(javaRDD.toString());;

and my map transformation is

public class Transformation implements Function<Tuple2<String, String>, String> {.........}

i want to pass the broadcast var to map function and do the join with other transformations.

Answer 1

The thing you are talking about is called Map-Side Join . In Spark it can be implemented using broadcast variable, here's a simple example in PySpark:

cities = {
        1 : 'Moscow',
        2 : 'London',
        3 : 'Paris',
        4 : 'Berlin',
        5 : 'New York'
    }
bcities = sc.broadcast(cities)

data = [
    [1, 1.23],
    [2, 2.34],
    [3, 3.45],
    [4, 4.23],
    [5, 24.24],
    [1, 32.2],
    [2, 22.2],
    [4, 222.3]
]
sc.parallelize(data).map(lambda x: [bcities.value[x[0]], x[1]]).collect()

If the dataset it bigger, it is better to implement Reduce-Side Join using Spark join() transformation

For Java see the example from Learning Spark -- start from line 134 where you can find the line:

final Broadcast<String[]> signPrefixes = sc.broadcast(loadCallSignTable());

Answer 2

Added a constructor for the same and passed the broadcast variable. public Transformation(Broadcast> val) { }