如何从文本中查找肯定和否定单词的总数？

Question

I want to find the total number of positive and negative words matched from a given text. I have list of positive words in positive.txt file and list of negative words in negative.txt file. If a word is matched from positive word list, then I want a simple integer variable where the value is incremented by 1, same for the negative matched word. From my given code I am getting a paragraph which is under @class=[story-hed] . This is the text which I want to compare with the list of positive and negative words as well as total count of words. My code is,

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from dawn.items import DawnItem

class dawnSpider(BaseSpider):
   name = "dawn"
   allowed_domains = ["dawn.com"]
   start_urls = [
       "http://dawn.com/"
   ]

   def parse(self, response):

      hxs = HtmlXPathSelector(response)      
      sites = hxs.select('//h3[@class="story-hed"]//a/text()').extract()
      items=[]

      for site in sites:
         item=DawnItem()
         item['title']=site
         items.append(item)
      return items

Answer 1

The standalone code below could do the trick:

from collections import Counter

def readwords( filename ):
    f = open(filename)
    words = [ line.rstrip() for line in f.readlines()]
    return words

positive = readwords('positive.txt')
negative = readwords('negative.txt')

paragraph = 'this is really bad and in fact awesome. really awesome.'

count = Counter(paragraph.split())

pos = 0
neg = 0
for key, val in count.iteritems():
    key = key.rstrip('.,?!\n') # removing possible punctuation signs
    if key in positive:
        pos += val
    if key in negative:
        neg += val

print pos, neg

Here is what I have in the two input files:

positive.txt:

good 
awesome

negative.txt:

bad
ugly

and the output is: 2 1

To implement this in scrapy, you might want to use an item pipeline http://doc.scrapy.org/en/latest/topics/item-pipeline.html

Answer 2

First you may want to read the files. Assuming you have a word per line you can read all the words with the following code:

postive = [l.strip() for l in open("possitive.txt")]

Once done, you can create a dict which will hold the word as key and the count as value. For initiating the dict to zero you can use:

positive_count = dict.fromkeys(postive, 0)

Finally you hust iterate all the items and increment the count if world is found:

for item in items:
    if item in positive_count:
         postive_count[item] +=1

And finally you can print the results with:

for item, value in postive_counts.iteritems():
    print "Word %s count %d" % (item, value)

For negative will be the same, just ommited to simplify the answer.

Answer 3

This depends on the size of the word lists. If they are smallish (less than a few kb), then read them into a list:

with open(positive_wordlist_file_name) as fd:
  positive_words = [line.strip() for line in fd]

Once you have two word lists, you can then got through the text with them - line by line if you can. Split those into words, and then use the "in" operator to check them in the list. I'd use a couple of co-routines in a class for it:

class WordCounter:
  # You can probably read word lists and store them here
  def positive_word_counter(self):
    """Co-routine that will count positive words. I'll leave it to reader
    to make a similar negative word one"""
    self.positive_words = 0
    while True:
      words = yield
      matched = [word for word in words if word in self.positive_words]
      self.positive_words += len(matched)

  def read_text(text):
    """Text - some iterable of lines - an file handle, or list or whatever."""
    #expand on this split with other word separators - or use re.split with the word boundary instead
    line_words = (line.strip().split(' ,') for line in text)
    #Create and prime coroutines
    positive_counter = self.positive_word_counter()
    positive_counter.next()
    negative_counter = self.negative_word_counter()
    negative_counter.next()
    #Now fire it in
    [[positive_counter.next(words), negative_counter.next(words)] for words in line_words]
    #You should now be able to read positive/negative words from this object

Answer 4

for key, val in count.iteritems(): ==>only it works in Python 3 below version if you're using python 3 above versions use

for key, val in count.item()
    key = key.rstrip('.,?!\n') # removing possible punctuation signs
    if key in positive:
        pos += val
    if key in negative:
        neg += val