Spark / Scala-在Hive表中加载数据框的更快方法

Question

I have a data frame which I intend to write as a Hive partitioned table. The code I use to that end is

for(i <- 0 until 10)
{
  myDf = hiveContext.sql("select * from srcTable where col = "+i)
  myDf.write.mode("append").format("parquet")
        .partitionBy("period_id").saveAsTable("myTable")
}

myDf will contain a different set of data in every iteration (I have just shown an oversimplified way of how I get values in myDf )

The myDf.write takes about 5 minutes to load 120,000 rows of data. Is there any way I could further reduce the time taken to write all this data?

Answer 1

first, why do you iterate and not just lead/save all data at once? Second, I could imagine that with your code, you write too many (small) files, you could check that on the filesystem. Normally I repartition my dataframe according to the same column which I use as partition-column of the DataFrameWriter , like this I get only 1 file per partition (as long as it isn't too large, otherwise HDFS will automatically split the file):

val cols = (0 until 10)

hiveContext.table("srcTable")
  .where($"col".isin(cols:_*))
  .repartition($"period_id")
  .write
  .format("parquet")
  .partitionBy("period_id")
  .saveAsTable("myTable")

Other than that, it's always a good idea to like into SparkUI and check whether the number of tasks is in a reasonable relation with the number of executors/cores.