将JSON数据添加到Scala中的多行字符串以使用Spark处理

Question

I am trying to utilize some parameters which are in multiline single json object in a json file stored on s3. However, because I am facing several issues for reading and parsing json in spark(honestly, its pain...), I tried using jackson converted a hardcoded multiline json to map as:

Following is my json hardcoded as multiline string:

val jsonString = 
    """
        {
          myJSON
        }
    """

I used jackson binder to decode it:

    val mapper = new ObjectMapper
    mapper.registerModule(DefaultScalaModule)
    mapper.readValue(jsonString, classOf[Map[String, String]])

Now I can use a map very easily. Unfortunately all the code base uses a map, hence this method seems preferable to me.

So I wanted to know if there is a way to create a multiline string with a json file in spark-scala? I will be fetching my json file from s3.

Answer 1

If you are not bounded by jackson, then you can try do it much easy and faster with jsoniter_scala . Add dependencies to your build script. Import and use them like here:

// import required packages
import java.io._
import com.github.plokhotnyuk.jsoniter_scala.macros._
import com.github.plokhotnyuk.jsoniter_scala.core._

// create JSON codec for your map
val codec = JsonCodecMaker.make[Map[String, String]](CodecMakerConfig())

// then read JSON file using it
val map = {
  val in: InputStream = // <- here can be any input stream implementation, no buffering required 
    new FileInputStream("/tmp/input.json")
  try JsonReader.read(codec, in)
  finally in.close()
}