pyspark 提取 json 值列並使用請求通過 rest 發布

Question

我在 PySpark 中有一個 dataframe 1 行 1 列 - json

-----------------------------------------------------------------------------------------
|json                                                              
-----------------------------------------------------------------------------------------
|[{"a":{"b":0,"c":{"50":0.005,"60":0,"100":0},"d":0.01,"e":0,"f":2}}]|
-----------------------------------------------------------------------------------------

我需要提取 json 值並使用請求通過 rest 發布它。

Answer 1

from pyspark.sql import SparkSession
import json
spark = (SparkSession.builder.appName("AuthorsAges").getOrCreate())
# Creating the DataFrame
data_df = spark.createDataFrame([["[{\"a\":{\"b\":0,\"c\": 
{\"50\":0.005,\"60\":0,\"100\":0},\"d\":0.01,\"e\":0,\"f\":2}}]"]])
data_df.show(1, False)
extract_text = data_df.collect()[0][0]
extract_json = json.loads(extract_text[1:-1])
# you can access any of the josn fields like this afterwards
print(extract_json['a']['c'])