连接条件期间的Pyspark内存问题

Question

I am using spark 2.1.0. I have 2 dataframe not more than 3 mb . When I tried to run inner join on 2 dataframes all my transformation logic every thing works perfectly. But when I use RightOuter join on 2 dataframes I get below error.

Error

RN for exceeding memory limits. 1.5 GB of 1.5 GB physical memory used. 
Consider boosting spark.yarn.executor.memoryOverhead.
17/08/02 02:29:53 ERROR cluster.YarnScheduler: Lost executor 337 on ip-172-
21-1-105.eu-west-1.compute.internal: Container killed by YARN for exceeding 
memory limits. 1.5 GB of 1.5 GB physical memory used. Consider boosting 
spark.yarn.executor.memoryOverhead.
17/08/02 02:29:53 WARN scheduler.TaskSetManager: Lost task 34.0 in stage 
283.0 (TID 11396, ip-172-21-1-105.eu-west-1.compute.internal, executor 337): 
ExecutorLostFailure (executor 337 exited caused by one of the running tasks) 
Reason: Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 
GB physical memory used. Consider boosting 
spark.yarn.executor.memoryOverhead.
17/08/02 02:29:53 WARN server.TransportChannelHandler: Exception in 
connection from /172.21.1.105:50342
java.io.IOException: Connection reset by peer

I tried with alternatives 1)df.coalesce(xvalue).show() 2)Tried setting executor memory nothing worked.

This issue is pending from past few weeks. Can anyone please let me know where am I going wrong

Answer 1

Could you please share the details regarding the datasets.

1. How many rows and columns in both datasets.

Have you tried leftOuterJoin, is it also giving you the same error.

Regards,

Neeraj