使用正则表达式从熊猫数据帧python中删除在单列的所有行中找到的唯一单词

Question

I am passing about 11,000 texts extracted from csv as a dataframe to the remove_unique function. I am finding unique words and saving it as list named as "unique" in the function. The unique words is created out of all the unique words found in the entire column.

Using regex I'm trying to remove the unique words from each row(single column) of panda dataframe but the unique words do not get removed as expected, instead all the words are removed and empty "text" is returned.

def remove_unique(text):
   //Gets all the unique words in the entire corpus
    unique = list(set(text.str.findall("\w+").sum()))
    pattern = re.compile(r'\b(?:{})\b'.format('|'.join(unique)))
    //Ideally should remove the unique words from the corpus.
    text = text.apply(lambda x: re.sub(pattern, '', x)) 
    return text

Can somebody tell point out what is the issue?

before
0    card layout broken window resiz unabl error ex...
1    chart lower rang border patch merg recheck...
2    left align text team close c...
3    descript sma...
4    list disappear navig make contain...
Name: description_plus, dtype: object
0
1                                                  ...
2
3
4                                                  ...
Name: description_plus, dtype: object

Answer 1

Not sure I totally understand. Are you trying to see if a word appears multiple times throughout the entire column?

Maybe

import re
a_list = list(df["column"].values) #column to list
string = " ".join(a_list) # list of rows to string
words = re.findall("(\w+)", string) # split to  single list of words

print([item for item in words if words.count(item) > 1]) #list of words that appear multiple times