在Spark Dataframe行上進行操作

Question

我是Spark，Scala等新手，下面是我的代碼

val eventdf = sqlContext.sql("SELECT sensor, data.actor FROM eventTable")

eventdf.map {
  case (r) => (r.getString(0) + count, r.getString(1), count)
}.saveToCassandra("caliper", "event", SomeColumns("sensor", "sendtime", "count"))

在這里，我想用r.getString(1)執行一些操作，然后傳遞給cassandra保存。

Answer 1

如果您不能將轉換直接應用於dataframe列，則可以提出以下建議：

import org.apache.spark.sql.Row
import sqlContext.implicits._

val newRDD = eventdf.map {
  case Row(val1: String, val2: String) => 
    // process val2 here and save the result to val2_processed
    (val1 + count, val2_processed, count) 
}

val newDF = newRDD.toDF("col1", "col2", "col3") // If you need to convert it back to DF

newDF.saveToCassandra(...)