[英]Spark parallelism with cores
I have started learning spark and while trying to run this example:我已经开始学习 spark 并在尝试运行此示例时:
package examples
import org.apache.spark.sql.SparkSession
object Test extends App {
val spark: SparkSession = SparkSession.builder()
.master("local[2]")
.appName("SparkByExample")
.getOrCreate()
println("First SparkContext:")
println("APP Name :"+spark.sparkContext.appName)
println("Deploy Mode :"+spark.sparkContext.deployMode)
println("Master :"+spark.sparkContext.master)
println("Default Min parallelism" + spark.sparkContext.defaultMinPartitions)
println("Default parallelism" + spark.sparkContext.defaultParallelism)
val sparkSession2: SparkSession = SparkSession.builder()
.master("local[1]")
.appName("SparkByExample-test")
.getOrCreate()
println("Second SparkContext:")
println("APP Name :"+sparkSession2.sparkContext.appName)
println("Deploy Mode :"+sparkSession2.sparkContext.deployMode)
println("Master :"+sparkSession2.sparkContext.master)
println("Default Min parallelism" + sparkSession2.sparkContext.defaultMinPartitions)
println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)
}
Here i have created two spark Session the first one with two cores and second one with one core, but when i run it, i get two parallelism for both sessions, I do not understand why?在这里,我创建了两个 spark Session,第一个有两个核心,第二个有一个核心,但是当我运行它时,两个会话都有两个并行性,我不明白为什么?
First SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2
Second SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2
println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)
"sparkContext.defaultParallelism" is returns default level of parallelism defined on “sparkContext.defaultParallelism”是返回定义的默认并行级别
SparkContext ,its default value calculated based on no of cores available on your application. SparkContext ,它的默认值是根据您的应用程序上可用的内核数计算的。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.