[英]Spark scala: SELECT in a foreach loop returns java.lang.NullPointerException
我需要在foreach循环中使用各种SELECT语句迭代DF的内容,将输出写入文本文件。 foreach循环中的任何SELECT语句都返回NullPointerException。 我无法理解为什么会这样。 “for”循环内的SELECT语句不会返回此错误。
这是测试用例。
// step 1 of 6: create the table and load two rows
vc.sql(s"""CREATE TEMPORARY TABLE TEST1 (
c1 varchar(4)
,username varchar(5)
,numeric integer) USING com.databricks.spark.csv OPTIONS (path "/tmp/test.txt")""")
// step 2 of 6: confirm that the data is queryable
vc.sql("SELECT * FROM TEST1").show()
+----+--------+-------+
| c1|username|numeric|
+----+--------+-------+
|col1| USER1| 0|
|col1| USER2| 1|
+----+--------+-------+
// Step 3 of 6: create a dataframe for the table
var df=vc.sql("""SELECT * FROM TEST1""")
// step 4 of 6: create a second dataframe that we will use as a loop iterator
var df_usernames=vc.sql(s"""SELECT DISTINCT username FROM TEST1 """)
// step 5 of 6: first foreach loop works ok:
df_usernames.foreach(t =>
{
println("(The First foreach works ok: loop iterator t is " + t(0).toString() )
}
)
(The First foreach works ok: loop iterator t is USER1
(The First foreach works ok: loop iterator t is USER2
// step 6 of 6: second foreach with any embedded SQL returns an error
df_usernames.foreach(t =>
{
println("(The second foreach dies: loop iterator t is " + t(0).toString() )
vc.sql("""SELECT c1 FROM TEST1""").show()
}
)
The second foreach dies: loop iterator t is USER1
org.apache.spark.SparkException: Job aborted due to stage failure: Task 158 in stage 94.0 failed 1 times, most recent failure: Lost task 158.0 in stage 94.0 (TID 3525, localhost): java.lang.NullPointerException
at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:195)
它无法完成。 如果不首先调用collect,则无法在foreach中启动SQL查询
>>> df_usernames.collect.foreach(
... lambda x: sqlContext.sql("""SELECT c1 FROM TEST1""").show())
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.