如何使用NLTK在CSV文件中查找特定单词的频率分布

Question

I am just starting in python and nltk and trying to read records from a csv file and determine the frequency of specific words across all records. I can do something like this:

with f:
    reader = csv.reader(f)

    # Skip the header
    next(reader)

    for row in reader:
        note = row[4]
        tokens = [t for t in note.split()] 

        # Calculate row frequency distribution
        freq = nltk.FreqDist(tokens) 
        for key,val in freq.items(): 
            print (str(key) + ':' + str(val))

        # Plot the results
        freq.plot(20, cumulative=False)

I am not sure how to modify this so that the frequency is across all records and that only the words that I am interested in are included. Apologies if this is a really simple question.

Answer 1

You can define the counter outside the loop freq_all = nltk.FreqDist() , then update it on each row freq_all.update(tokens)

with f:
    reader = csv.reader(f)

    # Skip the header
    next(reader)
    freq_all = nltk.FreqDist()

    for row in reader:
        note = row[4]
        tokens = [t for t in note.split()] 

        # Calculate raw frequency distribution
        freq = nltk.FreqDist(tokens) 
        freq_all.update(tokens)
        for key,val in freq.items(): 
            print (str(key) + ':' + str(val))

        # Plot the results
        freq.plot(20, cumulative=False)

    # Plot the overall results
    freq_all.plot(20, cumulative=False)