不能在Spark 1.5.2中使用超过5个数据框

Question

I would like to do more joins on multiple tables with pyspark,

and then I want to partition by date my tables:

my setup is as follows: 10GO MEMORY for driver 10 workers with 5 cores and 10G Memory yarn-customer

Table = 700 MO

Table2 = 1GB

Table 3 = 3Go

Table4 = 12

Table 4 go

table5 =6go

I tried

sqlcontext.sql ("select * from Tab1 left join tab2 is tab1.numcnt =      

tab2.numcnt  left join tab3 is tab1.numcnt = tab3.numcnt
left join TAB4 is tab1.numcnt = tab4.numcnt
")

when I use this query she takes a crazy time

I also tried the data frame methods:

df = join df_tab1.join (df_tab2, df_tab.NUMCNT df_tab2.NUMCNT == 'left_outer')

 dfjoin.join (df_tab3, df_join.NUMCNT df_tab3.NUMCNT == 'left_outer')

same problem 24 hours of treatment without result

if you can advise me how to properly join these thank you in advance

Answer 1

You need to cache all the tables which you are joining before performing the join. I would recommend to cache them, do a count on them and then perform join. I faced similar issue too but after performing join on cached tables, runtime gets significantly reduced.