任务不可序列化-正则表达式

Question

i have a movie which has a title. In this title is the year of the movie like "Movie (Year)". I want to extract the Year and i'm using a regex for this.

case class MovieRaw(movieid:Long,genres:String,title:String)
case class Movie(movieid:Long,genres:Set[String],title:String,year:Int)
val regexYear = ".*\\((\\d*)\\)".r
moviesRaw.map{case MovieRaw(i,g,t) => Movie(i,g,t,t.trim() match { case regexYear(y) => Integer.parseInt(y)})}

When executing the last command i get the following Error:

java.io.NotSerializableException: org.apache.spark.SparkConf

Running in the Spark/Scala REPL, with this SparkContext: val conf = new SparkConf(true).set("spark.cassandra.connection.host", "localhost") val sc = new SparkContext(conf)

Answer 1

As Dean explained, the reason of the problem is that the REPL creates a class out of the code added to the REPL and, in this case, the other variables in the same context are being "pulled" in the closure by the regex declaration.

Given the way you're creating the context, a simple way to avoid that serialization issue would be to declare the SparkConf and SparkContext transient:

@transient val conf = new SparkConf(true).set("spark.cassandra.connection.host", "localhost")
@transient val sc = new SparkContext(conf)

You don't even need to recreate the spark context in the REPL for the only purpose of connecting to Cassandra:

spark-shell --conf spark.cassandra.connection.host=localhost

Answer 2

You probably have this code in a larger Scala class or object (a type), right? If so, in order to serialize the regexYear , the whole enclosing type gets serialized, but you probably have the SparkConf defined in that type.

This is a very common and confusing problem and efforts are underway to prevent it, given the constraints of the JVM and languages on top of it, like Java.

The solution (for now) is to put regexYear inside a method or another object:

object MyJob {
  def main(...) = {
    case class MovieRaw(movieid:Long,genres:String,title:String)
    case class Movie(movieid:Long,genres:Set[String],title:String,year:Int)
    val regexYear = ".*\\((\\d*)\\)".r
    moviesRaw.map{case MovieRaw(i,g,t) => Movie(i,g,t,t.trim() match { case     regexYear(y) => Integer.parseInt(y)})}
    ...
  }
}

or

...
object small {
  case class MovieRaw(movieid:Long,genres:String,title:String)
  case class Movie(movieid:Long,genres:Set[String],title:String,year:Int)
  val regexYear = ".*\\((\\d*)\\)".r
  moviesRaw.map{case MovieRaw(i,g,t) => Movie(i,g,t,t.trim() match { case   regexYear(y) => Integer.parseInt(y)})}
}

Hope this helps.

Answer 3

Try passing in the cassandra option on the command line for spark-shell like this:

spark-shell [other options] --conf spark.cassandra.connection.host=localhost

And that way you won't have to recreate the SparkContext -- you can use the SparkContext (sc) that gets instantiated automatically with spark-shell.