将文本预处理函数应用于 scala spark 中的数据框列

Question

I want to create a function to handle the text-prepocessing in a problem I am facing with text data. I am familiar with Python and pandas dataframe and my usual thought process of solving the problem is to use a function and then using pandas apply method to apply the function to all the elements in a column. However I don't know where to begin to accomplish this.

So, I created two functions to handle the replacements. The problem is that I don't know how to put more than one replace inside this method. I need to make about 20 replacements for three separate dataframes so to solve it with this method it would take me 60 lines of code. Is there a way to do all the replacements inside a single function and then apply it to all the elements in a dataframe column in scala?

def removeSpecials: String => String = _.replaceAll("$", " ")
def removeSpecials2: String => String = _.replaceAll("?", " ")
val udf_removeSpecials = udf(removeSpecials)
val udf_removeSpecials2 = udf(removeSpecials2)
val consolidated2 = consolidated.withColumn("product_description", udf_removeSpecials($"product_description"))
val consolidated3 = consolidated2.withColumn("product_description", udf_removeSpecials2($"product_description"))
consolidated3.show()

Answer 1

Well you can simply add every replacement next to the previous one like this :

def removeSpecials: String => String = _.replaceAll("$", " ").replaceAll("?", " ")

But in this case where the replacement character is the same, it would be better to use regular expressions to avoid multiple replaceAll .

def removeSpecials: String => String = _.replaceAll("\\$|\\?", " ")

Note that \\\\ is used as escape character.