堆栈内存溢出
  简体   繁体   English

Spark 2.2 Null-safe Left Outer Join Null指针异常

[英]Spark 2.2 Null-safe Left Outer Join Null Pointer Exception

 原文  2017-10-10 14:12:31  1   1    scala/ apache-spark/ apache-spark-sql

问题描述

Performing a left outer join using the null-safe equals operator results in a NullPointerException . 使用null-safe equals运算符执行左外连接会导致NullPointerException 。

Versions Spark 2.2.0, Scala 2.11.8 版本 Spark 2.2.0，Scala 2.11.8 

scala> var d1 = Seq((null, 1), ("a1", 2)).toDF("a", "b")
scala> d1.show
+----+---+
|   a|  b|
+----+---+
|null|  1|
|  a1|  2|
+----+---+

scala> var d2 = Seq(("a2", 3)).toDF("a", "b")
scala> d2.show
+---+---+
|  a|  b|
+---+---+
| a2|  3|
+---+---+

scala> d1.joinWith(d2, d1("a") <=> d2("a"), "left_outer").show
17/10/10 09:44:39 ERROR Executor: Exception in task 0.0 in stage 6.0 (TID 8)
java.lang.NullPointerException

Is this behavior expected? 这种行为有望吗？

Stack trace 堆栈跟踪 

java.lang.NullPointerException
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
17/10/10 10:19:28 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.NullPointerException
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

17/10/10 10:19:28 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.NullPointerException
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

1 个解决方案

解决方案1
 0  2017-10-10 16:56:15

I can say no issue found in version Spark 2.2.0 and Scala 2.11.8 with same example, as i am not getting any exception when using same code example with null safe 我可以说在版本Spark 2.2.0和Scala 2.11.8中没有找到相同的例子，因为我在使用null安全的相同代码示例时没有得到任何异常

<=> <=>

and equal operator 和平等的运营商

=== ===

Can you check again and add more details related to issue ? 你能再次检查并添加与问题相关的更多细节吗？ 

val d1 = sc.parallelize(Seq(
          (null, 1), ("a1",2))
        ).toDF("a", "b") 

d1.show


+----+---+
|   a|  b|
+----+---+
|null|  1|
|  a1|  2|
+----+---+


val d2 = sc.parallelize(Seq(
      ("a2",3))
    ).toDF("a", "b") 

d2.show

+---+---+
|  a|  b|
+---+---+
| a2|  3|
+---+---+


d1.joinWith(d2, d1("a") <=>  d2("a"), "left_outer").show()

+--------+----+
|      _1|  _2|
+--------+----+
|[null,1]|null|
|  [a1,2]|null|
+--------+----+

d1.joinWith(d2, d1("a") ===  d2("a"), "left_outer").show()



+--------+----+
|      _1|  _2|
+--------+----+
|[null,1]|null|
|  [a1,2]|null|
+--------+----+

Adding other example : 添加其他示例： 

val x = Seq((100L,null), (102L,"17179869185L"), (101L,"17179869186L"), (200L,"17179869186L"), (401L,"1L"), (500L,"1L"), (600L,"8589934593L"), (700L,"8589934593L"), (800L,"8589934593L"), (900L,"8589934594L"), (1000L,"8589934594L"), (1200L,"2L"), (1300L,"2L"), (1301L,"2L"), (1400L,"17179869187L"), (1500L,"17179869188L"), (1600L,"8589934595L")).toDF("u","x1")

x.show()


+----+------------+
|   u|          x1|
+----+------------+
| 100|        null|
| 102|17179869185L|
| 101|17179869186L|
| 200|17179869186L|
| 401|          1L|
| 500|          1L|
| 600| 8589934593L|
| 700| 8589934593L|
| 800| 8589934593L|
| 900| 8589934594L|
|1000| 8589934594L|
|1200|          2L|
|1300|          2L|
|1301|          2L|
|1400|17179869187L|
|1500|17179869188L|
|1600| 8589934595L|
+----+------------+

val y = Seq(("17179869187L",-8589934595L), ("17179869188L",-8589934595L), ("17179869185L",-858993
4593L)).toDF("x2","y")


y.show()

+------------+-----------+
|          x2|          y|
+------------+-----------+
|17179869187L|-8589934595|
|17179869188L|-8589934595|
|17179869185L|-8589934593|
+------------+-----------+


x.join(y,'x1 === 'x2, "left_outer").show()

+----+------------+------------+-----------+
|   u|          x1|          x2|          y|
+----+------------+------------+-----------+
| 100|        null|        null|       null|
| 102|17179869185L|17179869185L|-8589934593|
| 101|17179869186L|        null|       null|
| 200|17179869186L|        null|       null|
| 401|          1L|        null|       null|
| 500|          1L|        null|       null|
| 600| 8589934593L|        null|       null|
| 700| 8589934593L|        null|       null|
| 800| 8589934593L|        null|       null|
| 900| 8589934594L|        null|       null|
|1000| 8589934594L|        null|       null|
|1200|          2L|        null|       null|
|1300|          2L|        null|       null|
|1301|          2L|        null|       null|
|1400|17179869187L|17179869187L|-8589934595|
|1500|17179869188L|17179869188L|-8589934595|
|1600| 8589934595L|        null|       null|
+----+------------+------------+-----------+

x: org.apache.spark.sql.DataFrame = [u: bigint, x1: string]
y: org.apache.spark.sql.DataFrame = [x2: string, y: bigint]
Command took 1.00 second

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Null值Spark中的左外部联接 - Null value Left Outer Join in Spark 在spark数据帧离开外连接后用null替换空值 - Replacing null values with 0 after spark dataframe left outer join Play Framework 2.2空指针异常ArrayList - Play Framework 2.2 Null Pointer Exception ArrayList 在Spark结构流2.3.0中连接两个流时，左外部连接不发出空值 - Left outer join not emitting null values when joining two streams in spark structured streaming 2.3.0 如何在 SPARK 数据帧 v1.6 的左外连接中将 NULL 替换为 0 - How to replace NULL to 0 in left outer join in SPARK dataframe v1.6 播放2.x无效安全合并（避免模板中的NPE） - Play 2.x null-safe coalescing (avoiding NPEs in templates) 如何在 Scala 中编写适当的空安全合并运算符？ - How to write a proper null-safe coalescing operator in scala? Scala Slick：如何进行除联接（左外部联接为null）？ - Scala Slick: How to do an except join (left outer join with is null)? Spark程序Map函数中的空指针异常 - Null Pointer exception in Map function of Spark program 在火花 UDF 中获取 Null 指针异常 - Getting Null Pointer exception in spark UDF
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM