从Python中的网页中删除停用词

Question

I tried for the below program its work properly:

I want to remove stopwords from the web pages, so the FEED_URL = ' http://feeds.feedburner.com/oreilly/radar/atom ' it run successfully but when i change the url then it will give an error

import os

import sys
import json
import feedparser
from BeautifulSoup import BeautifulStoneSoup
from nltk import clean_html

FEED_URL = 'http://feeds.feedburner.com/oreilly/radar/atom'            

def cleanHtml(html):
   return BeautifulStoneSoup(clean_html(html),
            convertEntities=BeautifulStoneSoup.HTML_ENTITIES).contents[0]

   fp = feedparser.parse(FEED_URL)

   print "Fetched %s entries from '%s'" % (len(fp.entries[0].title), fp.feed.title)
   #print "Fetched %s entries from '%s'" % (len(fp.entries[0])

   blog_posts = []
   for e in fp.entries:
      blog_posts.append({'title': e.title, 'content'
                  : cleanHtml(e.content[0].value), 'link': e.links[0].href})

      out_file = os.path.join('resources', 'ch05-webpages', 'feed.json')
      f = open(out_file, 'w')
      f.write(json.dumps(blog_posts, indent=1))
      f.close()
      print ('Wrote output file to %s' % (f.name, ))

But when i change the url then it gives Error

      FEED_URL = 'http://www.thehindu.com'

Error:

     IndexError                                Traceback (most recent call last)
     <ipython-input-1-b80b4061a360> in <module>()
     14 fp = feedparser.parse(FEED_URL)
     15 
     ---> 16 print "Fetched %s entries from '%s'" % (len(fp.entries[0].title), fp.feed.title)
     17 #print "Fetched %s entries from '%s'" % (len(fp.entries[0])
     18 

     IndexError: list index out of range

So anybody can help me for solving this problem?

Answer 1

Looks like the feed URL which you are using is not correct.

Try:

FEED_URL = 'http://www.thehindu.com/?service=rss'

For other feeds: http://www.thehindu.com/navigation/?type=rss