从python的计数器中删除停用词列表

Question

I've got a function in NLTK to generate a concordance list, which would look like

concordanceList = ["this is a concordance string something", 
               "this is another concordance string blah"] 

and I have another function which returns a Counter dictionary with the counts of each word in the concordanceList

def mostCommonWords(concordanceList):
  finalCount = Counter()
  for line in concordanceList:
    words = line.split(" ")
    currentCount = Counter(words)
    finalCount.update(currentCount)
  return finalCount

The problem I have is how best to remove stopwords from the resulting Counter, so that, when I call

mostCommonWords(concordanceList).most_common(10)

the result isn't just {"the": 100, "is": 78, "that": 57}.

I think that pre-processing the text to remove stopwords is out, because I still need the concordance strings to be instances of grammatical language. Basically, I'm asking if there's a simpler way to do this than creating a stopwords Counter for stopwords, setting the values low, and then making yet another Counter like so:

stopWordCounter = Counter(the=1, that=1, so=1, and=1)
processedWordCounter = mostCommonWords(concordanceList) & stopWordCounter

which should set the count values for all stopwords to 1, but it seems hacky.

Edit: Additionally, I'm having trouble actually making such a stopWordCounter, because if I want to include reserved words like "and", I get an invalid syntax error. Counters have easy to use union and intersection methods, which would make the task fairly simple; are there equivalent methods for dictionaries?

Answer 1

You can remove the stop words during tokenization...

stop_words = frozenset(['the', 'a', 'is'])
def mostCommonWords(concordanceList):
    finalCount = Counter()
    for line in concordanceList:
        words = [w for w in line.split(" ") if w not in stop_words]
        finalCount.update(words)  # update final count using the words list
    return finalCount

Answer 2

First, you don't need to create all those new Counter s inside your function; you can do:

for line in concordanceList:
    finalCount.update(line.split(" "))

instead.

Second, a Counter is a kind of dictionary, so you can delete items directly:

for sword in stopwords:
    del yourCounter[sword]

It doesn't matter whether sword is in the Counter - this won't raise an exception regardless.

Answer 3

I'd go for flattening the items into words, ignoring any stop words and providing that as input to a single Counter instead:

from collections import Counter
from itertools import chain

lines = [
    "this is a concordance string something", 
    "this is another concordance string blah"
]

stops = {'this', 'that', 'a', 'is'}    
words = chain.from_iterable(line.split() for line in lines)
count = Counter(word for word in words if word not in stops)

Or, that last bit can be done as:

from itertools import ifilterfalse
count = Counter(ifilterfalse(stops.__contains__, words))

Answer 4

You have a couple of options.

One, don't count the stopwords when updating your Counter - which you can do more concisely, since Counter objects can accept an iterable as well as another mapping for update :

def mostCommonWords(concordanceList):
    finalCount = Counter()
    stopwords = frozenset(['the', 'that', 'so'])
    for line in concordanceList:
        words = line.strip().split(' ')
        finalCount.update([word for word in words if word not in stopwords])
    return finalCount

Alternatively, you can use del to actually remove them from the Counter once you're done.

I've also added a strip call on line prior to split . If you were to use split() and the default behavior of splitting on all whitespace, you wouldn't need that, but split(' ') will not consider newline to be something to split on, so the last word of each line would have a trailing \\n and would be considered distinct from any other appearances. strip gets rid of that.

Answer 5

How about:

if 'the' in counter:
    del counter['the']

Answer 6

Personally, i think @JonClements' answer was the most elegant. BTW, there is already a list of stopwords in NLTK, just in case the OP didn't know, see NLTK stopword removal issue

from collections import Counter
from itertools import chain
from nltk.corpus import stopwords

lines = [
    "this is a concordance string something", 
    "this is another concordance string blah"
]

stops = stopwords.words('english')
words = chain.from_iterable(line.split() for line in lines)
count = Counter(word for word in words if word not in stops)
count = Counter(ifilterfalse(stops.__contains__, words))

Also, the FreqDist module in NLTK has more NLP related features as compared to collections.Counter . http://nltk.googlecode.com/svn/trunk/doc/api/nltk.probability.FreqDist-class.html