Extract feeds from web page

Question

I'm looking for a code snippet (language is not important here) that will extract all feeds (RSS, atom etc.) that is associated with this page.

So input is URL and output list of channels .

Important is completeness, it means if the page has associated some information channel it should be found.

I'm asking preferably for what to find in HTML code and where to find to cover completeness.

thank you

Answer 1

You find feeds in the head tag in html files. There they should be specified as link tags with an associated content type and a href attribute specifying it's location.

To extract all feed urls from a page using python you could use something like this:

import urllib
from HTMLParser import HTMLParser

class FeedParser(HTMLParser):

    def __init__(self, *args, **kwargs):
        self.feeds = set()
        HTMLParser.__init__(self, *args, **kwargs)

    def handle_starttag(self, tag, attrs): 
        if tag == 'link':
            try:
                href = [attr[1] for attr in attrs if attr[0] == 'href'][0]
            except IndexError:
                return None         
            else:
                if ('type', 'application/atom+xml') in attrs or ('type', 'application/rss+xml') in attrs:
                    self.feeds.add(href)    


def get_all_feeds_from_url(url):
    f = urllib.urlopen(url)
    contents = f.read()
    f.close()

    parser = FeedParser()
    parser.feed(contents)
    parser.close()

    return list(parser.feeds)

This code would have to be extended quite a bit though if you want to cover all the quirky ways a feed can be added to a html page.