线程“主”org.apache.spark.sql.catalyst.parser.ParseException 中的异常

Question

我有以下 dataframe：

dataframe1
+-----------------------+
|ID                     |
+-----------------------+
|[10,80,60,]            |
|[20,40,]               |
+-----------------------+

还有另一个 dataframe：

dataframe2
+------------------+----------------+
|ID_2              |   name         |
+------------------+----------------+
|40                | XYZZ           |
|200               | vbb            |
+------------------+----------------+

我想要以下 output：

+------------------+----------------+
|ID_2              |   name         |
+------------------+----------------+
|40                | XYZZ           |
+------------------+----------------+

我正在使用以下代码从第二个dataframe行中的 select 女巫 ID_2 == ID。

for (java.util.Iterator<Row> iter = dataframe1.toLocalIterator(); iter.hasNext();) {
        String item = (iter.next()).get(0).toString();
        dataframe2.registerTempTable("data2");
        Dataset<Row> res = sparkSession.sql("select * from data2 where ID_2 IN ("+item+")");
        res.show();
}

但我得到以下异常：

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
mismatched input 'from' expecting <EOF>(line 1, pos 9)

 == SQL ==
select * from data2 where ID_2 IN ([10,80,60,])
 ---------^^^

at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
at factory.Geofencing_Alert.check(Geofencing_Alert.java:84)
at factory.Geofencing_Alert.main(Geofencing_Alert.java:158)

我怎样才能解决这个问题？

Answer 1

只需使用explode function。

df1.withColumn("ID", explode($"ID"))
  .join(df2, $"ID" === $"ID_2", "inner")
  .drop("ID")
  .show

+----+----+
|ID_2|name|
+----+----+
|  40|xyzz|
+----+----+