[英]Pyspark replace strings in Spark dataframe column
I'd like to perform some basic stemming on a Spark Dataframe column by replacing substrings. 我想通过替换子字符串在Spark Dataframe列上执行一些基本的词干。 What's the quickest way to do this?
最快的方法是什么?
In my current use case, I have a list of addresses that I want to normalize. 在我目前的用例中,我有一个我想要规范化的地址列表。 For example this dataframe:
例如,这个数据帧:
id address
1 2 foo lane
2 10 bar lane
3 24 pants ln
Would become 会成为
id address
1 2 foo ln
2 10 bar ln
3 24 pants ln
For Spark 1.5 or later, you can use the functions package: 对于Spark 1.5或更高版本,您可以使用函数包:
from pyspark.sql.functions import *
newDf = df.withColumn('address', regexp_replace('address', 'lane', 'ln'))
Quick explanation: 快速解释:
withColumn
is called to add (or replace, if the name exists) a column to the data frame. withColumn
以向数据框添加(或替换,如果名称存在)列。 regexp_replace
will generate a new column by replacing all substrings that match the pattern. regexp_replace
将通过替换与模式匹配的所有子字符串来生成新列。 For scala 对于斯卡拉
import org.apache.spark.sql.functions.regexp_replace
import org.apache.spark.sql.functions.col
data.withColumn("addr_new", regexp_replace(col("addr_line"), "\\*", ""))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.