Spark：如何将 spark arrayType 作为表达式进行迭代

Question

Building a recursive function.

def loop(path: String, dt: DataType, acc:Seq[String]): Seq[String] = {
  dt match {
  case s: ArrayType => 
       s.fields.flatMap(f => loop(path + "." + f.name, f.dataType, acc))
  case s: StructType =>      
    s.fields.flatMap(f => loop(path + "." + f.name, f.dataType, acc))
  case other => 
    acc:+ path
}

I have an error saying that "error: value fields is not a member of org.apache.spark.sql.types.ArrayType". So how do I iterate over each element of the arrayType and return a flattened sequence of strings?

Answer 1

the trick is using .elementType

def loop(path: String, dt: DataType, acc:Seq[String]): Seq[String] = {
  dt match {
  case s: ArrayType =>
       loop(path, s.elementType, acc)
  case s: StructType =>      
    s.fields.flatMap(f => loop(path + "." + f.name, f.dataType, acc))
  case other => 
    acc:+ path
}