如何使用R使用流式XML（RSS提要）？

Question

I understand somewhat how to use the XML package to read and parse an XML file, such as a piece of an RSS feed. However, what is the basic setup for continuously reading an RSS feed?

For example, imagine that I want to set up a facility that continuously reads the feed from http://evemaps.dotlan.net/feed/sovereignty and stores the data in some kind of R data structure (say, a data.frame ). I imagine that I would need to do something like the following:

1. Set up R on a server (eg RStudio Server on an AWS instance)
2. Open a HTTP connection to the rss feed
3. Continuously read and parse distinct bits of the feed and add them to a data.frame which grows by each entry added

However, this is still a rather vague pictures. What are the basic packages and functions that I would need to string together to make this work? Meaning: what are the basic steps that I would need to put in place to create such a facility? I'm not looking for anyone to write this facility for me (even though that would be nice!). Rather, I'm trying to understand which overall steps are involved.

Answer 1

I think you're looking for pubsubhubbub .

With an RSS client (ie, your R application on AWS) you have 2 choices: polling or PubSubHubbub (aka webhooks, PuSH, and others). As mentioned here , with polling you may be throttled after exceeding some publisher's maximum-pings policy. With PuSH the publisher's server notifies your R application in realtime when there is a new update because it works as a subscription.

The SO answer linked above leads to the blog of popular pay-as-you-go hub provider, Superfeedr, and a post which describes the PuSH protocol's workflow and shows a command line implementation.

You can hear more about the protocol from this Google IO 2010 presentation by one of the engineers who crafted PuSH.