将JSON对象转换为Spark Java中的各个列

Question

I have a Spark Dataset and want to convert it into individual columns.

Using Spark 2.2 and java 1.8

DF.printSchema()
root
 |-- ute.internal.id: string (nullable = false)
 |-- ute.features.serialized: string (nullable = false)

DF.show()

{"ute.id":"123","ute.isBoolean":"true","ute.sortPriority":"5"},
{"ute.id":"456","ute.isBoolean":"false","ute.sortPriority":"6"}

Expected output - 
===============
ute.id|ute.feature.isBoolean|ute.sortPriority
123   |true                 |5 
456   |false                |6 

Someone can help on this?.Thanks.

Answer 1

val newDf = sqlContext.read.json(df.rdd)

It will give you a dataframe with all json columns

Example

  val json2 ="""{"ute.id":"123","ute.isBoolean":"true","ute.sortPriority":"5"},
              |{"ute.id":"456","ute.isBoolean":"false","ute.sortPriority":"6"}"""
  val jsonRdd = sc.parallelize(Seq(json2))
  val sqlContext = new SQLContext(sc)


  val df = sqlContext.read.json(jsonRdd)
  df.show(false)

+------+-------------+----------------+
|ute.id|ute.isBoolean|ute.sortPriority|
+------+-------------+----------------+
|123   |true         |5               |
+------+-------------+----------------+