使用 python 和 Counters 计算一个单词在文本中出现的次数

Question

I have a dictionary and a file with a lot of text (IMDB reviews). I am trying to update the dictionary with the number of times it has seen a word. I can get it to count how many letters are in the text file but I need the dictionary to have a count of the words instead. This is my current code:

import glob
import codecs
word_counts = Counter() # I require this

for label in [POS_LABEL, NEG_LABEL]:
    for directory in [TRAIN_DIR, TEST_DIR]:
        for fn in glob.glob(directory + "/" + label + "/*txt"):
            doc = codecs.open(fn, 'r', 'utf8') # Open the file with UTF-8 encoding
            k = doc.read().split()
            print(k)

print(word_counts)

Here is the result of this:

Counter({' ': 11507297, 'e': 6036511, 't': 4565527, 'a': 3979283, 'o': 3754514, 'i': 3654084, 's': 3342252, 'n': 3321375, 'r': 3003149, 'h': 2701649, 'l': 2194831, 'd': 1715235, 'c': 1351529, 'u': 1345564, 'm': 1317735, 'f': 1085018, 'y': 1031725, 'g': 1016124, 'w': 936183, 'b': 929635, 'p': 824208, '.': 650520, 'v': 617921, ',': 544818, 'k': 414662, 'I': 264480, "'": 263760, 'T': 220030, '/': 215720, '>': 202250, '<': 202094, '-': 132038, '"': 131858, 'A': 129582, 'S': 119546, 'B': 89001, 'x': 84005, 'M': 83700, 'H': 79548, 'C': 77492, 'D': 76272, 'j': 74962, ')': 71268, 'W': 70424, '(': 69585, 'E': 62149, 'R': 59387, 'L': 55859, 'O': 54998, 'N': 51707, 'P': 49662, '!': 49164, 'F': 47474, 'G': 47367, 'J': 41145, 'z': 40445, '0': 37357, 'q': 37098, '1': 35792, '?': 32338, 'V': 29518, 'K': 28837, 'Y': 21969, ':': 19800, '9': 19392, 'U': 17488, '2': 15978, '*': 13916, ';': 13375, '3': 11002, '5': 10457, '8': 8874, '4': 8342, '7': 8277, '&': 7714, '6': 6209, 'Z': 4490, 'é': 3337, 'Q': 2842, '\x96': 2529, 'X': 1957, '`': 1861, '$': 1617, '\x85': 1479, '_': 997, '%': 867, '+': 642, '#': 640, '=': 623, '\x97': 596, '´': 434, ']': 254, '’': 254, '[': 239, '~': 230, 'á': 208, '{': 192, '}': 192, '@': 181, 'è': 169, 'ö': 160, '–': 149, 'ó': 126, '\x91': 121, '£': 117, 'ü': 106, '\t': 106, 'í': 100, '^': 95, 'ä': 91, 'ç': 82, 'à': 80, 'ñ': 78, 'ô': 66, '¨': 64, 'ï': 58, '“': 57, '”': 55, '»': 55, '«': 53, 'ã': 48, 'â': 45, '|': 45, '\xa0': 44, '¡': 43, '½': 39, 'å': 36, 'ê': 36, '\\': 35, 'ë': 32, '\x84': 31, '·': 29, 'ú': 23, 'ý': 22, 'ø': 19, '\x8e': 18, '\x9e': 18, '‘': 18, '\x95': 17, '…': 16, '¦': 14, '§': 13, 'É': 10, 'ß': 10, 'î': 9, '\x80': 8, 'ð': 8, 'Æ': 8, 'Õ': 7, '\uf0b7': 7, 'Á': 6, 'ì': 6, 'æ': 6, 'Ü': 6, 'û': 6, 'ù': 6, 'ò': 5, '\xad': 5, 'Ö': 5, '、': 5, '\x08': 4, '°': 4, '®': 4, 'ō': 4, '¾': 4, 'Ã': 3, '¿': 3, 'À': 3, 'Å': 3, 'Ó': 3, '\x8d': 3, '¤': 2, 'Ê': 2, '₤': 2, 'Ä': 2, 'È': 2, 'Þ': 2, '，': 2, '¢': 2, 'º': 2, '▼': 2, '★': 2, '³': 1, '\x9a': 1, 'Ø': 1, 'Ï': 1, 'Â': 1, 'Ç': 1, 'Ð': 1, 'ı': 1, 'ğ': 1, '″': 1, '©': 1, 'ª': 1, '\x10': 1, 'Ż': 1, 'י': 1, 'ג': 1, 'א': 1, 'ל': 1, 'כ': 1, 'ר': 1, 'מ': 1, 'ו': 1, 'ן': 1, 'õ': 1})

Forgot to mention I tried something like this too:

k = doc.read().split()
word_counts.update(k)

inside of the third for loop.

I was told I was in the right direction:

if word_counts["movie"] == 61492

UPDATE:

for label in [POS_LABEL, NEG_LABEL]:
    for directory in [TRAIN_DIR, TEST_DIR]:
        for fn in glob.glob(directory + "/" + label + "/*txt"):
            doc = codecs.open(fn, 'r', 'utf8') # Open the file with UTF-8 encoding
            k = doc.read().split()
            word_counts.update(k)


print(word_counts["movie"])

This is where I am at now. This prints "movie" 60762 so I am a bit confused how I am short of 61492.

Answer 1

from collections import Counter
c = Counter()
print ('Initial :', c)
c.update('abcdaab')
print ('Sequence:', c)
c.update({'a':1, 'd':5}) #Update with this dictionary in Counter
c.update('zz') #Updates this string in Counter
print ('Updated Value    :', c)

###################### You can use any of the method ##################
'''
Output of above Code:
Initial : Counter()
Sequence: Counter({'a': 3, 'b': 2, 'c': 1, 'd': 1})
Updated Value    : Counter({'d': 6, 'a': 4, 'b': 2, 'z': 2, 'c': 1})
'''
def word_count(words):
    counts = dict()
    for word in words:
        if word in counts:
            counts[word] += 1
        else:
            counts[word] = 1
    return counts

print(word_count(["foo","bar","foo","foo","bar"])) #Prints {'bar': 2, 'foo': 3}
#If you want to maintain the order, use Ordered Dict

Is this what you are looking for ? Pass the list and get unique count of words in dictionary or do you want to use Counter Specifically ? This serves the same purpose as well.

Answer 2

I Think this is what you are looking for. Tell me if it helps.

from collections import Counter

count = Counter(["car","van","van","car","van"])
print(dict(count)) # for each word and its occurrence 
print(count['van']) # for specific word and its occurrence

Answer 3

I think you're nearly there. I would also strip punctuation characters at each side of a word and convert the word to lower case. So 'Movie' and 'movie' would be equivalent.

Try this:

import string
from collections import Counter

word_counts = Counter()

for label in [POS_LABEL, NEG_LABEL]:
    for directory in [TRAIN_DIR, TEST_DIR]:
        for fn in glob.glob(directory + "/" + label + "/*txt"):
            doc = codecs.open(fn, 'r', 'utf8') # Open the file with UTF-8 encoding
            k = [word.strip(string.punctuation) for word in doc.read().split()]
            word_counts.update(k)
            doc.close()  #remember to close the file