获取数据框中最常见（常见）单词的平均分数

Question

I am trying to get the average scores for the most common words in my dataframes. Currently my dataframe has this format.

sentence            |    score
"Sam I am Sam"      |      10
"I am Sam"          |      5
"Paul is great Sam" |      5
"I am great"        |      0 
"Sam Sam Sam"       |      15

I managed to successfully get the most common words using this blurp of code. This cleaned up my dataframe and removed all stop words. Which yielded me this series.

from collections import Counter

nltk.download('stopwords')
df_text = df[['sentence','score']]
df_text['sentence'] = df_text['sentence'].replace("[a-zA-Z0-9]{14}|rt|[0-9]",'',regex=True, inplace=False)
df_text['sentence'] = df_text['sentence'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
top_words =pd.Series(' '.join(df_text['sentence']).lower().split()).value_counts()[:25]
Words     |    Freq
Sam       |     7
I         |     3
Am        |     3 
Great     |     2
is        |     1

I understand that the groupby.().mean() is a really important function I would need to use, but I dont understand how I would try to get the score column. This is the ideal output I am trying to get. I showed the math to give logic on how I got the averages.

Words     |    Avg
Sam       |     35/7 = 5
I         |     15/3 = 5
Am        |     15/3 = 5
Great     |     5/2 = 2.5
is        |     5/1 = 5

Answer 1

I will skip the data cleaning part (such as stopword removal), except that you really should use nltk.word_tokenize instead of split() . In particular, it would be your responsibility to eliminate the quotes.

df['words'] = df['sentence'].apply(nltk.word_tokenize)

Once the words are extracted, count them and combine with the scores:

word_counts = pd.concat([df[['score']],
                         df['words'].apply(Counter).apply(pd.Series)], 
                        axis=1)

Now, calculate the weighted sums:

ws = word_counts.notnull().mul(word_counts['score'], axis=0).sum() \
                                               / word_counts.sum()
#score    1.0
#``       7.0
#Sam      5.0
#I        5.0
#am       5.0
#''       7.0
#Paul     5.0
#is       5.0
#great    2.5

Finally, eliminate the first row that was included only for convenience:

del(ws['score'])

Answer 2

considering you have your data in a tabular format.. this should work

import pandas as pd
from collections import Counter

df = pd.read_csv('data.csv')
cnt = Counter([word for sen in df.sentence.values for word in sen.split()])

for item in cnt:
    tot_score = 0
    for row in df.iterrows():
        if item in row[1]['sentence'].split():
            tot_score += row[1]['score']
    if cnt[item] != 0:
        print(item, tot_score/cnt[item])
    else:
        print(item, 0)