I have a Spark Dataset and want to convert it into individual columns.
Using Spark 2.2 and java 1.8
DF.printSchema()
root
|-- ute.internal.id: string (nullable = false)
|-- ute.features.serialized: string (nullable = false)
DF.show()
{"ute.id":"123","ute.isBoolean":"true","ute.sortPriority":"5"},
{"ute.id":"456","ute.isBoolean":"false","ute.sortPriority":"6"}
Expected output -
===============
ute.id|ute.feature.isBoolean|ute.sortPriority
123 |true |5
456 |false |6
Someone can help on this?.Thanks.
val newDf = sqlContext.read.json(df.rdd)
It will give you a dataframe with all json columns
Example
val json2 ="""{"ute.id":"123","ute.isBoolean":"true","ute.sortPriority":"5"},
|{"ute.id":"456","ute.isBoolean":"false","ute.sortPriority":"6"}"""
val jsonRdd = sc.parallelize(Seq(json2))
val sqlContext = new SQLContext(sc)
val df = sqlContext.read.json(jsonRdd)
df.show(false)
+------+-------------+----------------+
|ute.id|ute.isBoolean|ute.sortPriority|
+------+-------------+----------------+
|123 |true |5 |
+------+-------------+----------------+
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.