dataframe 中特定行中的值的总和

Question

I have a dataframe called 'test' like this:

import pandas as pd
from collections import Counter
from nltk import ngrams

data = [['john tom hello text shine bright', 10], ['random text hello text shine bright', 15], ['random text hello text shine bright juli', 14], 
       ['random text hello great shine bright', 15], ['random text hello great shine bright juli', 14]]

df = pd.DataFrame(data, columns=['Text', 'Value'])
df

Text	Value
john tom hello text shine bright	34
random text hello text shine bright	42
random text hello text shine bright juli	42
random text hello great shine bright	42
random text hello great shine bright juli	42

I have the following code which looks for the most common phrase of 4 words, that looks like this:

vals_df_1 = [y for x in df['Text'] for y in x.split()]
c_fourgrams = Counter([' '.join(y) for x in [4] for y in ngrams(vals_df_1, x)])

df_1_fourgrams = pd.DataFrame({'ngrams': list(c_fourgrams.keys()),
                   'count': list(c_fourgrams.values())})

df_1_fourgrams = df_1_fourgrams.sort_values('count', ascending=False)
df_1_fourgrams = df_1_fourgrams.head()
df_1_fourgrams

Then the dataframe 'df_1_fourgrams' looks like this:

ngrams	count
hello text shine bright	3
shine bright random text	3
bright random text hello	3
random text hello great	2
text shine bright random	2

What I am missing is for each phrase to have a sum of the Value column. If it finds the phrase 'most common phrase is' in 5 rows, then I need to sum all of the values from the Value column in those 5 rows.

The resulting dataframe would look something like this:

ngrams	count	Value sum
hello text shine bright	3	118
shine bright random text	3	118
bright random text hello	3	118
random text hello great	2	84
text shine bright random	2	84

Is this possible? How could I do this?

Answer 1

You can achieve everything with pandas from the original dataframe:

out = (df
 .assign(words=[[' '.join(x) for x in ngrams(s.split(), 4)]
                for s in df['Text']])
 .explode('ngrams')
 .groupby('ngrams')['Value']
 .agg(['count', 'sum'])
 .sort_values('count', ascending=False)
 .head(5)
)

output:

                          count  sum
ngrams                              
hello text shine bright       3   39
hello great shine bright      2   29
random text hello great       2   29
random text hello text        2   29
text hello great shine        2   29