使用pyspark分发到镶木地板的csv是否已分发？

Question

I have the following code snippet on an AWS EMR master node to convert a csv file to a parquet file.

%pyspark


csv_path = "s3://<bucket>/file.csv"
p_path = "s3://<bucket>/file.parquet"

df = sqlContext.read.csv(csv_path, header=True, inferSchema=True)
df.write.parquet(p_path, mode='overwrite')

If I request more nodes, will this operation be faster? In other words, is the conversion to parquet distributed in a spark cluster. I can't tell yet and I don't want to burn money on more nodes without knowing a little more about it.

Answer 1

Yes, it is distributed.

Will the operation be faster? It depends on many factors, but in the best case it should scale linearly in terms of number of nodes as long as the code is equivalent to the one you (single stage job).

Another improvement to disable schema inference and providing explicit schema.