Setting spark.driver.maxResultSize in EMR

Question

I am trying to set the spark.driver.maxResultSize in my pyspark job. I tried setting the conf setting inside my pyspark script like so:

spark.conf.set("spark.driver.maxResultSize", "4g")

But in my Spark environment it still shows the default 2G

I tried setting it in my AWS EMR spark-submit options with both --spark.driver.maxResultSize 4g and --conf spark.driver.maxResultSize 4g I get unrecongnized option as an error.

Am I missing the correct syntax?

Answer 1

我猜语法是--conf spark.driver.maxResultSize=4g而不是--conf spark.driver.maxResultSize 4g这似乎有效并且在 Spark 环境中正确显示。