如何将JSON文件的一部分加载到DataFrame？

Question

I have a file that has contents like this:

a {"field1":{"field2":"val","field3":"val"...}}
b {"field1":{"field2":"val","field3":"val"...}}
...

and I was able to load the file to a table like this:

╔════╦════════════════════════════════════════════════
║ ID ║  JSON                                         ║
╠════╬════════════════════════════════════════════════
║  a ║ {"field1":{"field2":"val","field3":"val"...}} ║
║  b ║ {"field1":{"field2":"val","field3":"val"...}} ║
╚════╩════════════════════════════════════════════════

How can I make it into something like this?

╔════╦═════════════════════════════════════
║ ID ║ field2  ║field3 ║...     ║...     ║
╠════╬═════════════════════════════════════
║  a ║ val     ║val    ║..      ║...     ║
║  b ║ val     ║val    ║..      ║...     ║
╚════╩═════════════════════════════════════

Since it is a partial json file, I cannot do read.json I saw this post too convert lines of json in RDD to dataframe in apache Spark But my json string is a nested json and it is very long, so I do not want to list out all the fields. I also tried

#solr_data is the data frame made from the file, and json is the column with the json string, session is a SparkSession
json_table = solr_data.select(solr_data["json"]).rdd.map(lambda x:session.read.json(x))

That did not work well. I can't show() nor collect() for that, createDataFrame() didn't work for that either.

Answer 1

使用select("JSON.field1.*")将子JSON“解构”为列。