计算每个单词出现的不同行数

Question

I have a Pandas DataFrame (or a Series, given that I'm just using one column) that contains strings. I also have a list of words. For each word in this list, I want to check how many different rows it appears in at least once. For example:

words = ['hi', 'bye', 'foo', 'bar']
df = pd.Series(["hi hi hi bye foo",
                "bye bye bye bye",
                "bar foo hi bar",
                "hi bye foo bar"])

In this case, the output should be

0   hi      3
1   bye     3
2   foo     3
3   bar     2

Because "hi" appears in three different rows (1st, 3rd and 4th), "bar" appears in two (3rd and 4th), and so on.

I came up with the following way to do this:

word_appearances = {}
for word in words:
    appearances = df.str.count(word).clip(upper=1).sum()
    word_appearances.update({word: appearances})

pd.DataFrame(word_appearances.items())

This works fine, but the problem is that I have a rather long list of words (around 40,000), around 30,000 rows to check and strings that are not as short as the ones I used in the example. When I try my approach with my real data, it takes forever to run. Is there a way to do this in a more efficient way?

Answer 1

Try list comprehension and str.contains and sum

df_out = pd.DataFrame([[word, sum(df.str.contains(word))] for word in words], 
                       columns=['word', 'word_count'])

Out[58]:
  word  word_count
0   hi           3
1  bye           3
2  foo           3
3  bar           2

Answer 2

word_appearances = {}
for word in words:
    appearances = df.str.count(word).clip(upper=1).sum()
    word_appearances[word]= appearances

pd.DataFrame.from_dict(word_appearances,columns=['Frequency'],orient='index')