使用Scala在Spark中对多列执行相同的操作

Question

I'm new in scala and Spark and wanted to try some simple concurrent operation on a matrix.

I have an [m, 2] matrix and want to divide each element of a column for the last element of that column.

Here an example of what I want to achieve:

 9   25        3  5
 27  10    ->  9  2
 6   15        2  3
 3   5         1  1

I can do this whit a simple for-loop, but wanted to do the operation on the columns simultaneously. It is possible in spark or is better to use scala concurrent?

Answer 1

The most important question here is what is your data volume? Spark is designed to be used on large amounts of data, too large to be processed or even stored on one computer. If you are wondering if to do something in Spark or on a single machine in plain scala then you should probably stop considering using Spark (if your data volume is not going to grow in the future).

Anyway, assuming for the moment that you are going to have large amounts of data you could do:

import spark.implicits._

val df = Seq(
  (1, 9, 25),
  (2, 27, 10),
  (3, 6, 15),
  (4, 3, 5)
).toDF("id", "n1", "n2")

val lastRow = df.orderBy(col("id").desc).first()

val result = df.withColumn("n1", col("n1") / lastRow.getInt(1))
    .withColumn("n2", col("n2") / lastRow.getInt(2))

result.show()

result:

+---+---+---+
| id| n1| n2|
+---+---+---+
|  1|3.0|5.0|
|  2|9.0|2.0|
|  3|2.0|3.0|
|  4|1.0|1.0|
+---+---+---+

Please not that this is quite inefficient - even taking the last element is very costly here (not to mention the overhead of launching a Spark job). Doing something like this in Spark might be a good idea only when the data volume is large and you are forced to use cluster computing.

Answer 2

Here you go:

val df = Seq((9, 25), (27, 10), (6, 15), (3, 5)).toDF

        val df_final = df.columns.foldLeft(df) {(tempDF, colName) => {
            tempDF.withColumn(colName, (col(colName) / 
             lit(df.select(colName).collect.last.getInt(0))).cast("Int"))
        }}