如何在python的500个文本文件中找到500个最常用的单词？

Question

I have 500 text files in one directory.I have to find 500 most frequent words in all of the text files combined.How can I achieve that?

PS: I have searched a lot but could not find a solution.

Answer 1

Use collections.Counter :

import os
from collections import Counter

c, directory = Counter(), 'path_to_your_directory'

for x in os.listdir(directory):
    fname = os.path.join(directory, x)
    if os.path.isfile(fname):
        with open(fname) as f:
            c += Counter(f.read().split())

for word, _ in c.most_common(500):
    print(word)

Of course, it will read every file found in that directory. If that's not the intended behavior, use glob.glob or glob.iglob with the required pattern instead of os.listdir (see Reut's comment to my answer).

Answer 2

This is the most straightforward way I could think of using a dictionary for the count, with the key as the word ad the value for the count:

import os
# word counts are stored in a dictionary
# for fast access and duplication prevention
count = {}
# your text files should be in this folder
DIR = "files"
# iterate over all files in the folder
for filename in os.listdir(DIR):
    with open(os.path.sep.join([DIR, filename]), 'r') as f:
        for line in f.readlines():
            # strip line separators from end of line
            line = line.strip()
            # once we have a line from the file, split it to words, and
            # add word to the scores (if it's new), or increase it's count
            for word in line.split():
                if word in count:
                    # existing
                    count[word] = count[word] + 1
                else:
                    # new
                    count[word] = 1
print sorted(count.items(), key=lambda x: x[1], reverse=True)[:500]

Using collections.Counter (as Padraic suggested):

import os
from collections import Counter

count = Counter()
DIR = "files"
for filename in os.listdir(DIR):
    with open(os.path.sep.join([DIR, filename]), 'r') as f:
        for line in f.readlines():
            line = line.strip()
            # count all words in line
            count.update(line.split())
print count.most_common(500)

Answer 3

You could create a counter for each new word, and an array of words. Add each New word to the array. Compare each word In the text file(s) to the words in the array using "index of", increment the counter for the word. Or you could create one array, populate with every NEW word from the text files, use second element of the array as a counter.

Answer 4

We can use Counter method from collections module.

1. Read only text files from target directory by glob
2. Iterate all files from step 1 by for loop .
3. Open file in read mode by with statement and read() method of file object.
4. Split content of file by split() method of string and use Counter to create countable dictionary. Add add two counters together. https://docs.python.org/2/library/collections.html
5. Get most common word from the Counter by most_common(3) method.

code:

from glob import glob 
from  collections import Counter

p = "/home/vivek/Desktop/test/*.txt"
main_counter = Counter()

for i in glob(p):
    with open(i, "rb") as fp:
        main_counter += Counter(fp.read().split())

print "main_counter:-", main_counter
print "most common 3:-", main_counter.most_common(3)

output:-

vivek@vivek:~/Desktop$ python 4.py 
main_counter:- Counter({'This': 3, 'try': 2, 'again.': 2, 'is': 2, 'can': 2, 'file': 2, 'you': 2, 'my': 2, '1': 1, 'this': 1, '2': 1})
most common 3:- [('This', 3), ('try', 2), ('again.', 2)]