过滤掉停用词

Question

I've created a simple word count program and I'm trying to filter out commonly used words from my list using nltk (see below).

My question is how would I apply my "stop" filter to my "frequency" list?

#Start 
from nltk.corpus import stopwords
import re
import string
frequency = {}
document_text = open('Import.txt', 'r')
text_string = document_text.read().lower()
match_pattern = re.findall(r'\b[a-z]{3,15}\b', text_string)

for word in match_pattern:
    count = frequency.get(word,0)
    frequency[word] = count + 1

frequency = {k:v for k,v in frequency.items() if v>1}

stop = set(stopwords.words('english'))
stop = list(stop)
stop.append(".")

import csv

with open('Export.csv', 'w') as csvfile:
    writer = csv.writer(csvfile)
    for key, value in frequency.items():
       writer.writerow([key, value])

Answer 1

stop = set(stopwords.words('english'))
stop.(".")

frequency = {k:v for k,v in frequency.items() if v>1 and k not in stop}

While stop is still a set , check the keys of your frequency dictionary when doing the comprehension. You can still make stop a list again afterwards.

The reason I keep it as a set is because it is much more efficient to search sets than it is to search lists.