Scala —转换和映射数据

Question

The raw data file I will be reading is a tab-delimited and one of the fields is a timestamp:

timestamp  userId  keyword
1405377264  A      google
1405378945  B      yahoo
1405377264  C      facebook

I got a case class defined as:

case class Event(date: String, userId: Int, keyword: String)

How do I go about transforming the timestamp to Date format and then map to the Event case class? I have the logic to convert the timestamp to Date:

import java.text.SimpleDateFormat
import java.util.Date

val df = new SimpleDateFormat("yyyy-MM-dd")
val dt = new Date(timestamp*1000L)
val date = df.format(dt) 

What is the right way to convert the raw data and map it to the case class?

Thanks!

Answer 1

I don't know if I'd say this is the right way, but one way to read each line would be to use regex extraction. Assuming you already have the data as a string, each line tab delimited, and each line separated by a line feed ( \\n ):

val data: String = ...
val regex = "(\\d+)\t([A-z])\t([A-z]+)".r

data.split('\n').map { line =>
    val regex(timestamp, userId, keyword) = line 
    Event(df.format(new Date(timestamp.toLong*1000L), userId, keyword)
}

As is, this is not fault tolerant if there are any deviations from the regex (which would have to be tweaked to your needs, I only followed the example above to the letter). If for example you wanted to discard the lines that didn't conform, you could use Try and collect :

data.split('\n').map { line =>
    Try {
       // same as above
    }
}.collect {
    case Success(event) => event
}

Answer 2

How about reading the CSV file using scala.io.Source.fromFile(myFile.csv).getLines ? This should return an Iterator[String] which is a lazy collection!

You can map over each line to create an Event. But what you want is to convert the timestamp to a java.util.Date as a first step before you create your Event objects.

I would suggest something along these lines: (This may not compile but it should give you the basic idea)

scala.io.Source.fromFile(myFile.csv).getLines flatMap { line =>
  splitAtDelimiter(line).toList match {
  case ts :: id :: kw :: Nil => 
    val date: Option[String] = 
      try { Some(convertToDateString(ts)) } catch { case e: Throwable => None }
    date.map(Event(_, getUserIdColumn(line), getKeyWordColumn(line)) // returns an Option[Event]
  case _ => None
}

where your convertToDateString would be a function which takes the timestamp value and converts it to a java.util.Date and then does a toString on it (looking at what you need for the date type in Event case class) and the splitAtDelimiter is an imaginary CSV parser function!