Spark Java 中的内部连接

Question

I'm trying to join two files 10 GB each and using inner join. In my cluster there are 3 worker nodes.

Here is my configuration

Driver Memory: 5G
Executor Memory: 15 G
Cores/Executor: 1
spark.cores.max: 9

Both files are joining on the basis of ID column. There are 10 millions of records in each file.

Its taking never ending time.

Can anyone please suggest ?

Answer 1

You can try following options:

1. Re-partition data

2. Fix skewed data issue if exists

3. set spark.sql.shuffle.partitions (otherwise it will create 200 partitions by default)

4. Update Spark Configuration (considering one node is of - 15G RAM) and 9 Cores

Driver Memory: 2G
Executor Memory: 7G
cores: 4