如何用 Bigram 计算 tfidf？

Question

在这里，我有一个名为 tweetRV.csv 的 dataframe，有 3 列（推文、停用词和群组）。 而且我只使用名称为（停用词）的列作为 TFIDF 并且我设法通过下面的代码获得了结果。 我已经设法将 Unigram 中的文本分开，然后像下面的代码一样计算 TFIDF

import pandas as pd
import numpy as np
from math import log

def idf(doc, wordBank):
N = len(doc.index)
result = pd.DataFrame(columns=['word', 'idf'])
for index, word in wordBank.iterrows():
  dft = np.sum(doc.str.contains(word['words']))
  idft = log(N / (dft + 1), 10)
  result = result.append(pd.Series([word['words'],  idft], index=['word', 'idf']), ignore_index=True)
return result
def tf(doc, wordBank):
wordList = doc.str.split(' ')
maxFt = [len(s)  for s in wordList]
result = pd.DataFrame()
for index, word in wordBank.iterrows():
  ft = np.add([s.count(word['words']) for s in wordList], 0)
  ftd = 1 + np.log10(ft + 1e-15)
  result = result.append(pd.Series(ftd), ignore_index=True)
result = result.replace(-np.inf, 0)
return result

def tfIdf(tf, idf):
result = pd.DataFrame()
for i in tf:
  tfIdf = tf[i] * idf['idf']
  result = result.append(pd.Series(tfIdf), ignore_index=True)
return result

print('Run it?')
print('Input (Y/N)')
option = input()
print('\n')

if(option == 'Y'):
print('Running...')
print('\n')
text = ""
df_csv = pd.read_csv('tweetRV.csv', encoding='utf-8')

for index, row in df_csv.iterrows():
text += row['Stopword'].replace('[','').replace("'","").replace(",","").replace("]"," ")

data = {'sentences':(text)}
doc = pd.DataFrame([data])

text = ""
itemText = []

df_csv = pd.read_csv('tweetRV.csv', encoding='utf-8')

for index, row in df_csv.iterrows():
text += row['Stopword'].replace('[','').replace("'","").replace(",","").replace("]"," ")

listText = list(text.split(' '))
del listText[-1]

for item in listText:
if item in itemText:
  pass
else:
  itemText.append(item)

data = {'words':itemText}
vocabulary = pd.DataFrame(data=data)

doc = doc['sentences']

resultIDF = idf(doc, vocabulary)
resultTF = tf(doc, vocabulary)

resultTfIdf = tfIdf(resultTF, resultIDF)
resultTfIdf.columns = itemText
resultTfIdf = resultTfIdf.transpose()

print(resultTfIdf)
resultTfIdf.to_csv('result.csv', encoding='utf-8')
else:
print('Closing...')

Stopword 列中的单词已经像这样被标记

Stopword
['red', 'velvet', 'seulgi', 'tells', 'sasaengs', 'stop', 'calling', 'live', 'instagram', 'broadcast']
['producers', 'red', 'velvet', 'psycho', 'reveal', 'song', 'addictive']
['red', 'velvet', 'voice', 'characters', 'dreamwork', 'trolls']

将这样的 TFIDF 结果作为 csv 文件

word     tfidf
red      -0.936380807003807
velvet   -0.933304531753474
wendy    -0.754125287111263
joy      -0.726979416468285
irene    -0.726979416468285

但现在我想用 Bigram 替换 Unigram 来做 TFIDF。 我想将 Bigram 中的文本分开，然后计算 TFIDF。 可能吗？ 这里有没有人可以帮助我的代码？ 谢谢你