Spark 與內核的並行性

Question

我已經開始學習 spark 並在嘗試運行此示例時：

package examples

import org.apache.spark.sql.SparkSession

object Test extends App {
  val spark: SparkSession = SparkSession.builder()
    .master("local[2]")
    .appName("SparkByExample")
    .getOrCreate()

  println("First SparkContext:")
  println("APP Name :"+spark.sparkContext.appName)
  println("Deploy Mode :"+spark.sparkContext.deployMode)
  println("Master :"+spark.sparkContext.master)
  println("Default Min parallelism" + spark.sparkContext.defaultMinPartitions)
  println("Default parallelism" + spark.sparkContext.defaultParallelism)

  val sparkSession2: SparkSession = SparkSession.builder()
    .master("local[1]")
    .appName("SparkByExample-test")
    .getOrCreate()

  println("Second SparkContext:")
  println("APP Name :"+sparkSession2.sparkContext.appName)
  println("Deploy Mode :"+sparkSession2.sparkContext.deployMode)
  println("Master :"+sparkSession2.sparkContext.master)
  println("Default Min parallelism" + sparkSession2.sparkContext.defaultMinPartitions)
  println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)
}

在這里，我創建了兩個 spark Session，第一個有兩個核心，第二個有一個核心，但是當我運行它時，兩個會話都有兩個並行性，我不明白為什么？

First SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2

Second SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2

Answer 1

   println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)

“sparkContext.defaultParallelism”是返回定義的默認並行級別
SparkContext ，它的默認值是根據您的應用程序上可用的內核數計算的。