Getting all facebook page wall posts and comments?

Question

I am developing a social media monitoring application. Currently, we are entering Facebook page ids into the application to collect data from possible customers' Facebook walls (so we have a realistic sample for the customer for direct promotion). These page ids are used to collect wall postings and comments and to compute statistics (eg to show most used words), and are presented to the user in a special view. Requirements are to collect all postings and comments without exception in near-live time. We currently have about 130 page ids in the system, with more to come.

Right now, I am using the Graph API for this, with several disadvantages:

• FB API access is restricted to 600 request/10 minutes. To get a near-live view, I need to access the API at least each two hours. As we are using API requests in other parts of the program, too, it is obvious that the limit is hit sooner or later (actually, this already happens)

• The responses are mostly redundant: to receive current comments, I have to request the wall postings (comments are enclosed in postings) with the URL http://graph.facebook.com/NAME/feed ...

• The probability for hitting the limits is dependent on the number of postings on the several walls

• I cannot get all comments with this method (eg comments on postings some time ago)

I am currently trying out how to switch to (or to complement Graph API usage) using FQL by querying the stream and the comment tables but this also has limitations:

• I cannot restrict my query to a specific timespan, leading to redundancy again

• The max number of posts I am getting for each one of my 130 page ids is 61 - (why 61?)

• I need an unpredictable number of additional requests because I need to get special objects like videos and links in separate requests.

My question now is - if anyone is doing similar things: How did you solve these problems? How do you get a pseudo-live-stream of a larger number (up to, say 1,000) of walls?

Letting the customer grant extra permissions to us is currently not an option.

Answer 1

you will probably have to meet with FaceBook and work out a contractual deal for greater access to their data. I would bet that the answer will be no, no and no, seeing as it appears you are trying to monetize their data, and furthermore, do so without the explicit permission of the users, but hey give it a shot.

Answer 2

I have a similar task - By default FB return only last ~50 posts or all in last 30 days(whichever is smaller) in FQL you should use created_time filter to receive more results. my current problem is that via FQL I receive no more than ~500 posts from any FB page wall even when LIMIT increased:

'select post_id from stream where source_id = 40796308305 and created_time <'.time().' LIMIT 1000000 ;'

this FQL request to CocaCola FB Page returns now ~300 posts only (less than 2 day posts).

If you find a better solution pls advise :)