在熊猫数据帧上计算IDF（反文档频率）

Question

I have a data frame df with three columns as shown below:

DocumentID    Words             Region
1             ['A','B','C']     ['Canada']
2             ['A','X','D']     ['India', 'USA', 'Canada']
3             ['B','C','X']     ['Canada']

I want to calculate IDF for each word in the "Words" column ie I want to generate an output which has each word like 'A','B','C' etc with its corresponding IDF value.

Answer 1

Here's a slightly less specific version. Assuming you want the standard 1/df definition of IDF, you can iterate through each "document" in the Words column counting:

from collections import defaultdict

# Assuming the Words column is represented as you presented it:
words = [['A','B','C'],
         ['A','X','D'],
         ['B','C','X']]

# to store intermediate counts:
idf = defaultdict(float)
for doc in words:
    for w in doc:
        idf[w] += 1

# Compute IDF as 1/df :
idf   = {k:(1/v) for (k,v) in idf.items()} #<- {'A': 0.5, 'B': 0.5,'C': 0.5, 'D': 1.0, 'X': 0.5}
vocab = idf.keys() # Note that the vocab is also accessible now.

Answer 2

list_words = []
list_regions = []

for words in df['Words']:

    for word in words:

        list_words.append(word)

for regions in df['Region']:

    for region in regions:

        list_regions.append(region)

IDF_words = pd.DataFrame([], columns=['words','IDF'])
IDF_regions = pd.DataFrame([], columns=['regions','IDF'])

IDF_words['words'] = sorted(set(list_words))
IDF_regions['regions'] = sorted(set(list_regions))

IDF_words['IDF'] = IDF_words['words'].map(lambda x: list_words.count(x)/float(len(list_words)))
IDF_regions['IDF'] = IDF_regions['regions'].map(lambda x: list_regions.count(x)/float(len(list_regions)))

hope it helps bro!
if it does pls upvote/mark answered :)
peace