简体   繁体   中英

Spark parallelism with cores

I have started learning spark and while trying to run this example:

package examples

import org.apache.spark.sql.SparkSession

object Test extends App {
  val spark: SparkSession = SparkSession.builder()
    .master("local[2]")
    .appName("SparkByExample")
    .getOrCreate()

  println("First SparkContext:")
  println("APP Name :"+spark.sparkContext.appName)
  println("Deploy Mode :"+spark.sparkContext.deployMode)
  println("Master :"+spark.sparkContext.master)
  println("Default Min parallelism" + spark.sparkContext.defaultMinPartitions)
  println("Default parallelism" + spark.sparkContext.defaultParallelism)

  val sparkSession2: SparkSession = SparkSession.builder()
    .master("local[1]")
    .appName("SparkByExample-test")
    .getOrCreate()

  println("Second SparkContext:")
  println("APP Name :"+sparkSession2.sparkContext.appName)
  println("Deploy Mode :"+sparkSession2.sparkContext.deployMode)
  println("Master :"+sparkSession2.sparkContext.master)
  println("Default Min parallelism" + sparkSession2.sparkContext.defaultMinPartitions)
  println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)
}


Here i have created two spark Session the first one with two cores and second one with one core, but when i run it, i get two parallelism for both sessions, I do not understand why?

First SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2

Second SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2
   println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)

"sparkContext.defaultParallelism" is returns default level of parallelism defined on
SparkContext ,its default value calculated based on no of cores available on your application.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM