如何在Spark数据集中创建TypedColumn并进行操作？

Question

I'm trying to perform an aggregation using mapGroups that returns a SparseMatrix as one of the columns, and sum the columns.

I created a case class schema for the mapped rows in order to provide column names. The matrix column is typed org.apache.spark.mllib.linalg.Matrix . If I don't run toDF before performing the aggregation ( select(sum("mycolumn") ) I get one type mismatch error ( required: org.apache.spark.sql.TypedColumn[MySchema,?] ). If I include toDF I get another type mismatch error: cannot resolve 'sum(mycolumn)' due to data type mismatch: function sum requires numeric types, not org.apache.spark.mllib.linalg.MatrixUDT . So what's the right way to do it?

Answer 1

It looks you struggle with at least two distinct problems here. Lets assume you have Dataset like this:

val ds = Seq(
  ("foo",  Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))), 
  ("foo",  Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0)))
).toDS

Selecting TypedColumn :

• using implicit conversions with $ :

   ds.select(col("_1").as[String]) 

• using oassql.functions.col :

   ds.select(col("_1").as[String]) 

Adding matrices:

• MLLib Matrix and MatrixUDT don't implement addition. It means you won't be able to sum function or reduce with +
• you can use third party linear algebra library but this is not supported in Spark SQL / Spark Dataset

If you really want to do it with Datsets you can try to do something like this:

ds.groupByKey(_._1).mapGroups(
  (key, values) => {
    val matrices = values.map(_._2.toArray)
    val first = matrices.next
    val sum = matrices.foldLeft(first)(
      (acc, m) => acc.zip(m).map { case (x, y) => x + y }
    )
    (key, sum)
})

and map back to matrices but personally I would just convert to RDD and use breeze .