如何使用 Scala 从 JSON 文件中读取字符串列表

Question

val df = spark.read.option("multiline", "true").json("/FileStore/tables/config-5.json")
    
df.show()

Output:

+--------------+-------------------+
|      List-col|            Matrics|
+--------------+-------------------+
|[number, word]|ApproxCountDistinct|
|[number, word]|       Completeness|
+--------------+-------------------+

Code:

for (row <- df.rdd.collect) {   
    var List_col =(row(0))
    var Matricsdynamic = row(1)
    List_col.foreach(c =>print(c) )

    //MatricsCal.ApproxCountDistinct_func(listofStr)
}     

As List-col is supposed to be a list of string I am getting WrappedArray(number, word)WrappedArray(number, word). I need list(String).

Answer 1

I assume you need get second element from List-col, is so you can get it:

import scala.collection.mutable
import spark.implicits._
val df = Seq(
  (List("24", "text1"), "metric1"),
  (List("12", "text2"), "metric2"),
  (List("53", "text2"), "metric3"),
  (List("13", "text3"), "metric4"),
  (List("64", "text4"), "metric5")
).toDF("List-col", "Matrics")
val result: Array[String] = df.map{
  row =>
    row.get(0) match {
      case t:mutable.WrappedArray[AnyRef] => t.last.toString
    }
}.collect()
println(result.mkString("Array(", ", ", ")")) // Array(text1, text2, text2, text3, text4)

Answer 2

You should be able to convert easily to a List of String, using toList method of WrappedArray .

Assuming your JSON file looks something like:

{"List-col": [9, "word1"], "Matrics": "ApproxCountDistinct"}
{"List-col": [10, "word2"], "Matrics": "Completeness"}

You can get back an array of records, each record being a List[String] .

import org.apache.spark.sql._
import org.apache.spark.sql.functions.col
val lists:Array[List[String]] = df.select(col("List-col")).collect.map(
                               (r: Row) => r.getAs[WrappedArray[String]](0).toList)