在 Spark SQL 中将多个小表与大表连接的最佳方法

Question

I'm doing a join multiples tables using spark sql. One of the table is very big and the others are small (10-20 records). really I want to replace values in the biggest table using others tables that contain pairs of key-value.

ie Bigtable:

| Col 1 | Col 2 | Col 3 | Col 4 | ....
--------------------------------------
| A1    | B1    | C1    | D1    | ....
| A2    | B1    | C2    | D2    | ....
| A1    | B1    | C3    | D2    | ....
| A2    | B2    | C3    | D1    | ....
| A1    | B2    | C2    | D1    | ....
.
.
.
.
.

Table2:

| Col 1 | Col 2 
----------------
| A1    | 1a    
| A2    | 2a    

Table3:

| Col 1 | Col 2 
----------------
| B1    | 1b    
| B2    | 2b  

Table3:

| Col 1 | Col 2 
----------------
| C1    | 1c    
| C2    | 2c  
| C3    | 3c

Table4:

| Col 1 | Col 2 
----------------
| D1    | 1d    
| D2    | 2d  

Expected table is

| Col 1 | Col 2 | Col 3 | Col 4 | ....
--------------------------------------
| 1a    | 1b    | 1c    | 1d    | ....
| 2a    | 1b    | 2c    | 2d    | ....
| 1a    | 1b    | 3c    | 2d    | ....
| 2a    | 2b    | 3c    | 1d    | ....
| 1a    | 2b    | 2c    | 1d    | ....
.
.
.
.
.

My question is; which is best way to join the tables . (Think that there are 100 or more small tables) 1) Collecting the small dataframes, to transforming it to maps, broadcasting the maps and transforming the big datataframe in one only step

bigdf.transform(ds.map(row => (small1.get(row.col1),.....)

2) Broadcasting the tables and making join using select method.

spark.sql("
       select * 
       from bigtable
       left join small1 using(id1) 
       left join small2 using(id2)")

3) Broadcasting the tables and Concatenate multiples joins

bigtable.join(broadcast(small1), bigtable('col1') ==small1('col1')).join...

Thanks in advance

Answer 1

You might do:

1. broadcast all small tables (automaticaly done by setting spark.sql.autoBroadcastJoinThreshold slightly superior to the small table number of rows)

2. run a sql query that join the big table such

   val df = spark.sql(" select * from bigtable left join small1 using(id1) left join small2 using(id2)")

EDIT: Choosing between sql and spark "dataframe" syntax: The sql syntax is more readable, and less verbose than the spark syntax (for a database user perspective.) From a developper perspective, dataframe syntax might be more readeble.

The main advantage of using the "dataset" syntax, is the compiler will be able to track some error. Using any string syntax such sql or columns name (col("mycol")) will be spotted at run time.

Answer 2

If the data in your small tables is less than the threshold size and physical files for your data is in parquet format then spark will automatically broadcast the small tables but if you are reading the data from some other data sources like sql, PostgreSQL etc. then some times spark does not broadcast the table automatically.

If you know that the tables are small sized and the size of table is not expected to increase ( In case of lookup tables) you can explicitly broadcast the data frame or table and in this way you can efficiently join a larger table with the small tables.

you can verify that the small table is getting broadcasted using the explain command on the data frame or you can do that from Spark UI also.

Answer 3

Best way, as already written in answers, to broadcast all small tables. And us can be done by SQL only - with hint:

val df = spark.sql("""
    select /*+ BROADCAST(t2, t3) */
        * 
    from bigtable t1
        left join small1 t2 using(id1) 
        left join small2 t3 using(id2)
""")