[英]PySpark parse Json using RDD and json.load
{
"city": "Tempe",
"state": "AZ",
...
"attributes": [
"BikeParking: True",
"BusinessAcceptsBitcoin: False",
"BusinessAcceptsCreditCards: True",
"BusinessParking: {'garage': False, 'street': False, 'validated': False, 'lot': True, 'valet': False}",
"DogsAllowed: False",
"RestaurantsPriceRange2: 2",
"WheelchairAccessible: True"
],
...
}
你好,我正在使用 PySpark,我正在嘗試輸出一個 (state, BusinessAcceptsBitcoin) 元組,目前我正在做:
csr = (dataset
.filter(lambda e:"city" in e and "BusinessAcceptsBitcoin" in e)
.map(lambda e: (e["city"],e["BusinessAcceptsBitcoin"]))
.collect()
)
但是這個命令失敗了。 如何獲得“BusinessAcceptsBitcoin”和“城市”字段?
您可以使用 Dataframe 和 UDF 來解析“屬性”字符串。
從您提供的示例數據來看,“屬性”似乎不是正確的 JSON 或 Dict。
假設“屬性”只是一個字符串,這里是一個使用數據框和 Udf 的示例代碼。
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *
spark = SparkSession \
.builder \
.appName("test") \
.getOrCreate()
#sample data
data=[{
"city": "Tempe",
"state": "AZ",
"attributes": [
"BikeParking: True",
"BusinessAcceptsBitcoin: False",
"BusinessAcceptsCreditCards: True",
"BusinessParking: {'garage': False, 'street': False, 'validated': False, 'lot': True, 'valet': False}",
"DogsAllowed: False",
"RestaurantsPriceRange2: 2",
"WheelchairAccessible: True"
]
}]
df=spark.sparkContext.parallelize(data).toDF()
用戶定義的函數來解析字符串
def get_attribute(data,attribute):
return [list_item for list_item in data if attribute in list_item][0]
注冊udf
udf_get_attribute=udf(get_attribute, StringType
數據框
df.withColumn("BusinessAcceptsBitcoin",udf_get_attribute("attributes",lit("BusinessAcceptsBitcoin"))).select("city","BusinessAcceptsBitcoin").show(truncate=False)
樣本輸出
+-----+-----------------------------+
|city |BusinessAcceptsBitcoin |
+-----+-----------------------------+
|Tempe|BusinessAcceptsBitcoin: False|
+-----+-----------------------------+
例如,您也可以使用相同的 udf 來查詢任何其他字段
df.withColumn("DogsAllowed",udf_get_attribute("attributes",lit("DogsAllowed"))).select("city","DogsAllowed").show(truncate=False)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.