[英]PySpark: Read nested JSON from a String Type Column and create columns
我在 PySpark 中有一個 dataframe,有 3 列 - json,日期和 object_id:
-----------------------------------------------------------------------------------------
|json |date |object_id|
-----------------------------------------------------------------------------------------
|{'a':{'b':0,'c':{'50':0.005,'60':0,'100':0},'d':0.01,'e':0,'f':2}}|2020-08-01|xyz123 |
|{'a':{'m':0,'n':{'50':0.005,'60':0,'100':0},'d':0.01,'e':0,'f':2}}|2020-08-02|xyz123 |
|{'g':{'h':0,'j':{'50':0.005,'80':0,'100':0},'d':0.02}} |2020-08-03|xyz123 |
-----------------------------------------------------------------------------------------
現在我有一個變量列表:[a.c.60, an60, ad, gh]。 我只需要從上述 dataframe 的 json 列中提取這些變量,並將這些變量添加為 dataframe 中的列及其各自的值。
所以最后,dataframe 應該是這樣的:
-------------------------------------------------------------------------------------------------------
|json |date |object_id|a.c.60|a.n.60|a.d |g.h|
-------------------------------------------------------------------------------------------------------
|{'a':{'b':0,'c':{'50':0.005,'60':0,'100':0},'d':0.01,...|2020-08-01|xyz123 |0 |null |0.01|null|
|{'a':{'m':0,'n':{'50':0.005,'60':0,'100':0},'d':0.01,...|2020-08-02|xyz123 |null |0 |0.01|null|
|{'g':{'h':0,'j':{'k':0.005,'':0,'100':0},'d':0.01}} |2020-08-03|xyz123 |null |null |0.02|0 |
-------------------------------------------------------------------------------------------------------
請幫助獲得此結果 dataframe。我面臨的主要問題是由於傳入的 json 數據沒有固定結構。 json 數據可以是嵌套形式的任何數據,但我只需要提取給定的四個變量。 我在 Pandas 中實現了這一點,方法是展平 json 字符串,然后提取 4 個變量,但在 Spark 中它變得越來越困難。
有兩種方法可以做到:
get_json_object
function ,如下所示:import pyspark.sql.functions as F
df = spark.createDataFrame(['{"a":{"b":0,"c":{"50":0.005,"60":0,"100":0},"d":0.01,"e":0,"f":2}}',
'{"a":{"m":0,"n":{"50":0.005,"60":0,"100":0},"d":0.01,"e":0,"f":2}}',
'{"g":{"h":0,"j":{"50":0.005,"80":0,"100":0},"d":0.02}}'],
StringType())
df3 = df.select(F.get_json_object(F.col("value"), "$.a.c.60").alias("a_c_60"),
F.get_json_object(F.col("value"), "$.a.n.60").alias("a_n_60"),
F.get_json_object(F.col("value"), "$.a.d").alias("a_d"),
F.get_json_object(F.col("value"), "$.g.h").alias("g_h"))
會給:
>>> df3.show()
+------+------+----+----+
|a_c_60|a_n_60| a_d| g_h|
+------+------+----+----+
| 0| null|0.01|null|
| null| 0|0.01|null|
| null| null|null| 0|
+------+------+----+----+
from_json
function將 JSON 轉換為結構,然后從結構中提取單個值——這可能比 JSON 路徑更高效:from pyspark.sql.types import *
import pyspark.sql.functions as F
aSchema = StructType([
StructField("c", StructType([
StructField("60", DoubleType(), True)
]), True),
StructField("n", StructType([
StructField("60", DoubleType(), True)
]), True),
StructField("d", DoubleType(), True),
])
gSchema = StructType([
StructField("h", DoubleType(), True)
])
schema = StructType([
StructField("a", aSchema, True),
StructField("g", gSchema, True)
])
df = spark.createDataFrame(['{"a":{"b":0,"c":{"50":0.005,"60":0,"100":0},"d":0.01,"e":0,"f":2}}',
'{"a":{"m":0,"n":{"50":0.005,"60":0,"100":0},"d":0.01,"e":0,"f":2}}',
'{"g":{"h":0,"j":{"50":0.005,"80":0,"100":0},"d":0.02}}'],
StringType())
df2 = df.select(F.from_json("value", schema=schema).alias('data')).select('data.*')
df2.select(df2.a.c['60'], df2.a.n['60'], df2.a.d, df2.g.h).show()
會給
+------+------+----+----+
|a.c.60|a.n.60| a.d| g.h|
+------+------+----+----+
| 0.0| null|0.01|null|
| null| 0.0|0.01|null|
| null| null|null| 0.0|
+------+------+----+----+
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.