Spark 数据框未使用工作人员

Question

I have a spark cluster with 3 worker nodes, when i try to load the csv file from hdfs it only utilizes the resources(cpu & memory) on the system where i load the csv via spark-shell (used master node)

Load dataframe

val df = spark.read.format("csv")
.option("header","true")
.load("hdfs://ipaddr:9000/user/smb_ram/2016_HDD.csv")

Do some operation on the dataframe

df.agg(sum("failure")).show

When i load csv system memory increases by 1.3 GB which is the hdfs file size & 100 % CPU usage. The workers were idling CPU near 0 % and no memory usage changes. Ideally i would expect all the heavy lifting to be done by worker which is not happening.

Answer 1

Set spark mode to cluster that should solve your problem. Looks like your job is running in Client mode.