与groupBy聚合后将pyspark数据帧保存为CSV文件

Question

I am learning pyspark and I am a bit confused on how to save a grouped dataframe as a csv file (assuming that for some reasons --eg RAM limitations-- I don't want to convert it first to a Pandas dataframe).

For a reproducible example:

import seaborn as sns
import findspark
findspark.init()
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.master('local') \
.appName('Data cleaning') \
.getOrCreate()
from pyspark.context import SparkContext
sc = SparkContext.getOrCreate()
from pyspark.sql.functions import *

mpg= sns.load_dataset('mpg')
mpg_sp = spark.createDataFrame(mpg)
mpg_grp = mpg_sp.groupBy('model_year', 'origin').avg('displacement', 'weight')

# The command below fails in the sense that it creates a folder with multiple  files in it rather than a single csv file as I would expect

mpg_grp.write.csv('mpg_grp.csv')

# By applying the collect method I get a list which can not be saved as a csv file

mpg_grp1 = mpg_grp.collect()
type(mpg_grp1)
list

Answer 1

Above answer is correct but results of its use are not quite good.
Of course you can use repartition(1) or coalesce(1) but it will cause transferring all your data to a single worker and will greatly slow down your code.
In order to avoid this, I would suggest you to partition data on one of your columns in your dataset. And then write simple code to get one file per partition:

cols = ["$name"]
mpg_grp.repartition(cols).write.partitionBy(cols).csv("$location")

Thus, the data will be partitioned between workers by one of your columns and you will get exactly one file per your partition (by date as an example).

Answer 2

Spark is a distributed framework. Therefore, the output in several files is a normal behavior... each worker will write it's part which results in several small files.

You can trick a bit the system using this command :

mpg_grp.coalesce(1).write.csv('mpg_grp.csv')

This will write only 1 file (but still in a folder which name is 'mpg_grp.csv').
Caution: It may be quite slow.