计算NLTK语料库中的不停词

Question

In python using NLTK how would I find a count of the number of non stop words in a document filtered by category?

I can figure out how to get the words in a corpus filtered by a category eg all the words in the brown corpus for category 'news' is:

text = nltk.corpus.brown.words(categories=category)

And separately I can figure out how to get all the words for a particular document eg all the words in the document 'cj47' in the brown corpus is:

text = nltk.corpus.brown.words(fileids='cj47')

And then I can loop through the results and count up the words that are not stopwords eg

stopwords = nltk.corpus.stopwords.words('english')
for w in text:    
    if w.lower() not in stopwords:
#found a non stop words

But how do I put it together so that I am filtering by category for a particular document? If I try to specify a category and a filter at the same time eg

 text = nltk.corpus.brown.words(categories=category, fields=’cj47’)

I get an error saying:

 ValueError: Specify fields or categories, not both

Answer 1

1. Get fileids for a category:

   fileids = nltk.corpus.brown.fileids(categories=category)

2. For each file, count the non-stopwords:

   for f in fileids: words = nltk.corpus.brown.words(fileids=f) sum = sum([1 for w in words if w.lower() not in stopwords]) print "Document %s: %d non-stopwords." % (f, sum)