How do I retrieve RSS Feeds based on a date range?

Question

How do I retrieve RSS Feeds based on a date range?

Specifically, how do I prepare the url so that I can get items that were published past a certain date?

www.pwop.com/feed.aspx?show=dotnetrocks&filetype=master&tags=Craftsmanship

Answer 1

Your questions is more related to the HTTP API of the site, not RSS it self.

RSS is a predefined XML data format.

Most RSS urls doesn't support filters and introduce simple URL which returns in RSS format the last X results (x is usually between 10 to 50 results).

Some URL allow to specify categories or Tags like in your example, so the reutrn RSS XML will contain only results from this tags.

If you don't want to miss results, you need to keep query the RSS URL every X minutes/hours depends on the update speeds of the results.

Other option is to contact the site and request a full API access or even to implement a feature to filter by date.

Answer 2

Not all websites support it, but maybe there is a solution that can work:

Websites usually have a sitemap.xml (or sitemap.xml.gz or sitemap.gz) file that contains all the urls in bulk or grouped in some way (eg, by category, tag, month). The sitemap.xml can contain links to additional xmls and so on.

The main sitemap is typically located in the root of the site (eg, https://news.bitcoin.com/sitemap.xml ), but you can find more information about sitemaps here: https://www.sitemaps.org/protocol.html .

If a website has such an xml file, perhaps processing it will make it easier to extract the needed information without any special site crawler or API.