How do I parse a deeply nested JSON document that may have some missing or extra fields using Scala?

Question

I have read other Scala JSON parsing questions but they all seem to assume a very basic document that is not deeply nested or of mixed types. Or they assume you know all of the members of the document or that some will never be missing.

I am currently using Jackson's Streaming API but the code required is difficult to understand and maintain. Instead, I'd like to use Jerkson to return an object representing the parsed JSON if possible.

Basically: I'd like to replicate the JSON parsing functionality that's so familiar to me in dynamic languages. I realize that's probably very wrong, and so I'm here to be educated.

Say we have a Tweet:

{
 "id": 100,
 "text": "Hello, world."
 "user": {
          "name": "Brett",
          "id": 200
         },
 "geo": {
         "lat": 10.5,
         "lng": 20.7 
        }
}

Now, the Jerkson Case Class examples make a lot of sense when you only want to parse out, say, the ID:

val tweet = """{...}"""
case class Tweet(id: Long)
val parsed = parse[Tweet](tweet)

But how do I deal with something like the Tweet above?

Some gotchas:

1. Some fields can be null or missing, for example "geo" above may be null , or one day they may drop a field and I don't want my parsing code to fail.
2. Some fields will be extra, or fields will be added and I want to be able to ignore them.

Answer 1

Lift json-scalaz is the best way to read and write JSON that I have come across. Docs explain it's usage pretty well here:

https://github.com/lift/framework/tree/master/core/json-scalaz

Answer 2

Of course right after I post this I find some help elsewhere. :)

I believe the "richer JSON example" towards the bottom of this post is a step in the right direction: http://bcomposes.wordpress.com/2012/05/12/processing-json-in-scala-with-jerkson/

Answer 3

Another alternatie using Lift-json could be:

package code.json

import org.specs2.mutable.Specification
import net.liftweb.json._

class JsonSpecs extends Specification {

  implicit val format = DefaultFormats

  val a = parse("""{
                  | "id": 100,
                  | "text": "Hello, world."
                  | "user": {
                  |          "name": "Brett",
                  |          "id": 200
                  |         },
                  | "geo": {
                  |         "lat": 10.5,
                  |         "lng": 20.7
                  |        }
                  |}""".stripMargin)

  val b = parse("""{
                  | "id": 100,
                  | "text": "Hello, world."
                  | "user": {
                  |          "name": "Brett",
                  |          "id": 200
                  |         }
                  |}""".stripMargin)


  "Lift Json" should{
    "find the id" in {
      val res= (a \ "id").extract[String]
      res must_== "100"
    }
    "find the name" in{
      val res= (a \ "user" \ "name").extract[String]
      res must_== "Brett"
    }
    "find an optional geo data" in {
      val res= (a \ "geo" \ "lat").extract[Option[Double]]
      res must_== Some(10.5)
    }
    "ignore missing geo data" in {
      val res= (b \ "geo" \ "lat").extract[Option[Double]]
      res must_== None
    }
  }
}

Note how when the geo data is missing on the val b, the parsing works just fine, expecting a None.

Or do you want to get case classes as the result?

For a case class example, see:

package code.json

import org.specs2.mutable.Specification
import net.liftweb.json._

class JsonSpecs extends Specification {

  implicit val format = DefaultFormats

  case class Root(id: Int, text: Option[String], user: Option[User], geo: Option[Geo])
  case class User(name: String, id: Int)
  case class Geo(lat: Double, lng: Double)



val c = parse("""{
                | "id": 100
                | "user": {
                |          "name": "Brett",
                |          "id": 200
                |         },
                | "geo": {
                |         "lng": 20.7
                |        }
                |}""".stripMargin)


  "Lift Json" should{
    "return none for geo lat data" in {
      val res= c.extract[Root].geo.map(_.lat)
      res must_== None
    }
  }
}