[英]Extract String from text pyspark
我有一個 pyspark 數據框:
例子:
text <String> | name <String> | original_name <String>
----------------------------------------------------------------------------
HELLOWORLD2019THISISGOOGLE | WORLD2019 | WORLD_2019
----------------------------------------------------------------------------
NATUREISVERYGOODFORHEALTH | null | null
----------------------------------------------------------------------------
THESUNCONTAINVITAMIND | VITAMIND | VITAMIN_D
----------------------------------------------------------------------------
BECARETOOURHEALTHISVITAMIND | OURHEALTH | OUR_/HEALTH
----------------------------------------------------------------------------
我想循環name
列並查看text
中是否存在name
值,如果是,我創建一個new_column
,將包含text
找到的name
值的original_name
值。 知道有時數據框列是null
。
例子:
在數據框示例的第 4 行中, text
包含來自name
列的 2 個值: [OURHEALTH, VITAMIND]
,我應該獲取它的original_name
值並將它們存儲在new_column
。
在第2行中, text
包含OURHEALTH
從name
列中,我應在存儲new_column
原始name
值發現==> [OUR_/HEALTH]
期待結果:
text <String> | name <String> | original_name <String> | new_column <Array>
------------------------------|------------------|---------------------------|----------------------------
HELLOWORLD2019THISISGOOGLE | WORLD2019 | WORLD_2019 | [WORLD_2019]
------------------------------|------------------|---------------------------|----------------------------
NATUREISVERYGOODFOROURHEALTH | null | null | [OUR_/HEALTH]
------------------------------|------------------|---------------------------|----------------------------
THESUNCONTAINVITAMIND | VITAMIND | VITAMIN_D | [VITAMIN_D]
------------------------------|------------------|---------------------------|----------------------------
BECARETOOURHEALTHISVITAMIND | OURHEALTH | OUR_/HEALTH | [OUR_/HEALTH, VITAMIN_D ]
-----------------------------------------------------------------------------|----------------------------
我希望我的解釋清楚。
我嘗試了以下代碼:
df = df.select("text", "name", "original_name").agg(collect_set("name").alias("name_array"))
for name_item in name_array:
df.withColumn("new_column", F.when(df.text.contains(name_item), "original_name").otherwise(None))
有人可以幫我嗎? 謝謝
一種簡單的解決方案是在原始 DataFrame 和僅具有name
列的派生 DataFrame 之間使用join
。 由於多行可以滿足連接條件,因此我們必須在連接后按原始列進行分組。
這是您輸入的詳細示例:
data = [
("HELLOWORLD2019THISISGOOGLE", "WORLD2019", "WORLD_2019"),
("NATUREISVERYGOODFOROURHEALTH", None, None),
("THESUNCONTAINVITAMIND", "VITAMIND", "VITAMIN_D"),
("BECARETOOURHEALTHISVITAMIND", "OURHEALTH", "OUR_ / HEALTH")
]
df = spark.createDataFrame(data, ["text", "name", "original_name"])
# create new DF with search words
# as it's the originl_name which interests us for the final list so we select it too
search_df = df.select(struct(col("name"), col("original_name")).alias("search_match"))
# join on df.text contains search_df.name
df_join = df.join(search_df, df.text.contains(search_df["search_match.name"]), "left")
# group by original columns and collect matches in a list
df_join.groupBy("text", "name", "original_name")\
.agg(collect_list(col("search_match.original_name")).alias("new_column"))\
.show(truncate=False)
輸出:
+----------------------------+---------+-------------+--------------------------+
|text |name |original_name|new_column |
+----------------------------+---------+-------------+--------------------------+
|HELLOWORLD2019THISISGOOGLE |WORLD2019|WORLD_2019 |[WORLD_2019] |
|THESUNCONTAINVITAMIND |VITAMIND |VITAMIN_D |[VITAMIN_D] |
|NATUREISVERYGOODFOROURHEALTH|null |null |[OUR_ / HEALTH] |
|BECARETOOURHEALTHISVITAMIND |OURHEALTH|OUR_ / HEALTH|[VITAMIN_D, OUR_ / HEALTH]|
+----------------------------+---------+-------------+--------------------------+
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.