为什么这个Scala代码会变慢？

Question

I'm running the following Scala code:

import scala.util.parsing.json._
import scala.io._

object Main {
        def jsonStringMap(str: String) =
                JSON.parseFull(str) match {
                        case Some(m: Map[_,_]) => m collect {
                                        // If this doesn't match, we'll just ignore the value
                                        case (k: String, v: String) => (k,v)
                                } toMap
                        case _ => Map[String,String]()
                }

        def main(args: Array[String]) {
                val fh = Source.fromFile("listings.txt")
                try {
                        fh.getLines map(jsonStringMap) foreach { v => println(v) }
                } finally {
                        fh.close
                }
        }
}

On my machine it takes ~3 minutes on the file from http://sortable.com/blog/coding-challenge/ . Equivalent Haskell and Ruby programs I wrote take under 4 seconds. What am I doing wrong?

I tried the same code without the map(jsonStringMap) and it was plenty fast, so is the JSON parser just really slow?

It does seem likely that the default JSON parser is just really slow, however I tried https://github.com/stevej/scala-json and while that gets it down to 35 seconds, that's still much slower than Ruby.

I am now using https://github.com/codahale/jerkson which is even faster ! My program now runs in only 6 seconds on my data, only 3 seconds slower than Ruby, which is probably just the JVM starting up.

Answer 1

A quick look at the scala-user archive seems to indicate that nobody is doing serious work with the JSON parser in the scala standard library.

See http://groups.google.com/group/scala-user/msg/fba208f2d3c08936

It seems the parser ended up in the standard library at a time when scala was less in the spotlight and didn't have the expectations it has today.

Answer 2

Use Jerkson . Jerkson uses Jackson which is always the fastest JSON library on the JVM (especially when stream reading/writing) large documents.

Answer 3

Using my JSON library , I get an almost instantaneous parse of both files:

import com.github.seanparsons.jsonar._
import scala.io.Source
def parseLines[T](file: String, transform: (Iterator[String]) => T): T = {
  val log = Source.fromFile(file)
  val logLines = log.getLines()
  try { transform(logLines) } finally { log.close }
}
def parseFile(file: String) = parseLines(file, (iterator) => iterator.map(Parser.parse(_)).toList)
parseFile("products.txt"); parseFile("listings.txt")

However, as someone mentioned, it would be more useful to just parse the whole thing as a JSONArray rather than have lots of individual lines as this does.