I have the following as a results of a collect
Array[org.apache.spark.sql.Row] = Array([16660,23517,23518,10,1,0])
How do I pull info from this Array? I need to check if any of the values is a 0
val df = Seq((1,2,3,4,5),(1,0,3,4,5),(1,2,3,4,0)).toDF()
df.show()
+---+---+---+---+---+
| _1| _2| _3| _4| _5|
+---+---+---+---+---+
| 1| 2| 3| 4| 5|
| 1| 0| 3| 4| 5|
| 1| 2| 3| 4| 0|
+---+---+---+---+---+
val arr = df.collect()
// array with boolens indicating if 0 exist in each column
val res = Array.fill(arr(0).length)(false)
// convert Array[Row] to Array[Array[Int]]
val arr2 = for( j <- 0 until arr.length) yield {for (i <- 0 until arr(j).length) yield {(arr(j).getInt(i))}}
arr2.foreach(s=>for(i <- 0 until s.length) if(s(i)==0) res(i)=true)
output:
res: Array[Boolean] = Array(false, true, false, false, true)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.