Scala Spark Map DataFrame缺少参数类型

Question

I am new to Spark and receiving an error in when I map a Dataframe.

I have a DStream and I want to transform it using a sql Dataframe to filter the data. The code is like this:

  val textDStream = ssc.textFileStream(inputPath)
  val activityStream = textDStream.transform(input => {
    input.flatMap { line =>
      val record = line.split("\\t")
      Some(Activity(record(0).toLong / MS_IN_HOUR * MS_IN_HOUR, record(1), record(2), record(3), record(4), record(5), record(6)))
      }
  })

activityStream.transform(rdd => {

    val df = rdd.toDF()

    df.registerTempTable("activity")
    val activityByProduct = sqlContext.sql("""SELECT
                                        product,
                                        timestamp_hour,
                                        sum(case when action = 'purchase' then 1 else 0 end) as purchase_count,
                                        sum(case when action = 'add_to_cart' then 1 else 0 end) as add_to_cart_count,
                                        sum(case when action = 'page_view' then 1 else 0 end) as page_view_count
                                        from activity
                                        group by product, timestamp_hour """)

    activityByProduct
      .map { r => ((r.getString(0), r.getLong(1)),
        ActivityByProduct(r.getString(0), r.getLong(1), r.getLong(2), r.getLong(3), r.getLong(4))
        )}

  }).print()

The problem here is that I receive the following error:

Error:(58, 18) missing parameter type .map { r => ((r.getString(0), r.getLong(1)),

activityByProduct
  .map { r => ((r.getString(0), r.getLong(1)),
    ActivityByProduct(r.getString(0), r.getLong(1), r.getLong(2), r.getLong(3), r.getLong(4))
    )}

I cannot seen where the type is missing. I have already tried to explicitilly set r => type. But it continues to return the error.

What could it be ?

Thanks in advance

Answer 1

It worked.

I had to convert the dataframe to rdd before executing the map:

activityByProduct.rdd
      .map { r =>
        ((r.getString(0), r.getLong(1)),
        ActivityByProduct(r.getString(0), r.getLong(1), r.getLong(2), r.getLong(3), r.getLong(4))
        )}

look at the .rdd after the activityByProduct

Answer 2

Yes, this works. You need to to convert this to rdd if it has to work. It works well in previous Spark versions but with 2.12 and above you will need this.

Answer 3

Hello I also encountered the same problem with you. When I added the import hiveCtx.implicits._ code to the next line of my code val hiveCtx = new HiveContext (sc) the error was removed.Because this code converts the RDD implicitly to a DataFrame. Hope can help you.

The complete code is posted below hoping to help you.

package spark.sparkSQL

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.hive.HiveContext

object sparksql2 {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("sparksql").setMaster("local")
    val sc = new SparkContext(conf)
    sc.setLogLevel("ERROR")

    val hiveCtx = new HiveContext(sc)
    import hiveCtx.implicits._        //  ImportType(hiveCtx.implicits)

    val input = hiveCtx.jsonFile("./inputFile")
    // Register the input schema RDD
    input.registerTempTable("tweets")
    hiveCtx.cacheTable("tweets")
    // Select tweets based on the retweetCount
    val topTweets = hiveCtx.sql("SELECT text, retweetCount FROM tweets ORDER BY retweetCount LIMIT 10")
    topTweets.collect().map(println(_))
    val topTweetText = topTweets.map(row => row.getString(0))
  }
}