將JSON對象轉換為Spark Java中的各個列

Question

我有一個Spark數據集，並希望將其轉換為單獨的列。

使用Spark 2.2和Java 1.8

DF.printSchema()
root
 |-- ute.internal.id: string (nullable = false)
 |-- ute.features.serialized: string (nullable = false)

DF.show()

{"ute.id":"123","ute.isBoolean":"true","ute.sortPriority":"5"},
{"ute.id":"456","ute.isBoolean":"false","ute.sortPriority":"6"}

Expected output - 
===============
ute.id|ute.feature.isBoolean|ute.sortPriority
123   |true                 |5 
456   |false                |6 

有人可以幫忙嗎？謝謝。

Answer 1

val newDf = sqlContext.read.json(df.rdd)

它將為您提供一個包含所有json列的數據框

例

  val json2 ="""{"ute.id":"123","ute.isBoolean":"true","ute.sortPriority":"5"},
              |{"ute.id":"456","ute.isBoolean":"false","ute.sortPriority":"6"}"""
  val jsonRdd = sc.parallelize(Seq(json2))
  val sqlContext = new SQLContext(sc)


  val df = sqlContext.read.json(jsonRdd)
  df.show(false)

+------+-------------+----------------+
|ute.id|ute.isBoolean|ute.sortPriority|
+------+-------------+----------------+
|123   |true         |5               |
+------+-------------+----------------+