[英]How create new column in Spark using Python, based on other column?
My database contains one column of strings.我的数据库包含一列字符串。 I'm going to create a new column based on part of string of other columns.我将根据其他列的部分字符串创建一个新列。 For example:例如:
"content" "other column"
The father has two dogs father
One cat stay at home of my mother mother
etc. etc.
I thought to create an array with words who interessed me.我想用我感兴趣的单词创建一个数组。 For example: people=[mother,father,etc.]例如:people=[mother,father,etc.]
Then, I iterate on column "content" and extract the word to insert on new column:然后,我迭代列“内容”并提取要插入新列的单词:
def extract_people(df):
column=[]
people=[mother,father,etc.]
for row in df.select("content").collect():
for word in people:
if str(row).find(word):
column.append(word)
break
return pd.Series(column)
f_pyspark = df_pyspark.withColumn('people', extract_people(df_pyspark))
This code don't work and give me this error on the collect():此代码不起作用,并在 collect() 上给我这个错误:
22/01/26 11:34:04 ERROR Executor: Exception in task 2.0 in stage 3.0 (TID 36)
java.lang.OutOfMemoryError: Java heap space
Maybe because my file is too large, have 15 million of row.也许是因为我的文件太大,有 1500 万行。 How I may make the new column in different mode?如何以不同的模式制作新列?
Using the following dataframe as an example以下面的 dataframe 为例
+---------------------------------+
|content |
+---------------------------------+
|Thefatherhas two dogs |
|The fatherhas two dogs |
|Thefather has two dogs |
|Thefatherhastwodogs |
|One cat stay at home of my mother|
|One cat stay at home of mymother |
|Onecatstayathomeofmymother |
|etc. |
|my feet smell |
+---------------------------------+
You can do the following您可以执行以下操作
from pyspark.sql import functions
arr = ["father", "mother", "etc."]
expression = (
"CASE " +
"".join(["WHEN content LIKE '%{}%' THEN '{}' ".format(val, val) for val in arr]) +
"ELSE 'None' END")
df = df.withColumn("other_column", functions.expr(expression))
df.show()
+---------------------------------+------------+
|content |other_column|
+---------------------------------+------------+
|Thefatherhas two dogs |father |
|The fatherhas two dogs |father |
|Thefather has two dogs |father |
|Thefatherhastwodogs |father |
|One cat stay at home of my mother|mother |
|One cat stay at home of mymother |mother |
|Onecatstayathomeofmymother |mother |
|etc. |etc. |
|my feet smell |None |
+---------------------------------+------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.