如何从命令行检查 Spark 配置？

Question

Basically, I want to check a property of Spark's configuration, such as "spark.local.dir" through command line, that is, without writing a program. Is there a method to do this?

Answer 1

There is no option of viewing the spark configuration properties from command line.

Instead you can check it in spark-default.conf file. Another option is to view from webUI.

The application web UI at http://driverIP:4040 lists Spark properties in the “Environment” tab. Only values explicitly specified through spark-defaults.conf, SparkConf, or the command line will appear. For all other configuration properties, you can assume the default value is used.

For more details, you can refer Spark Configuration

Answer 2

以下命令在控制台上打印您的 conf 属性

 sc.getConf.toDebugString

Answer 3

We can check in Spark shell using below command :

scala> spark.conf.get("spark.sql.shuffle.partitions")
res33: String = 200

Answer 4

Based on http://spark.apache.org/docs/latest/configuration.html . Spark provides three locations to configure the system:

• Spark properties control most application parameters and can be set by using a SparkConf object, or through Java system properties.

• Environment variables can be used to set per-machine settings, such the IP address, through the conf/spark-env.sh script on each node.

• Logging can be configured through log4j.properties.

I haven't heard about method through command line.

Answer 5

从 CLI 检查 spark 配置的主命令

sc._conf.getAll()