通过用户登录从其他网站导入用户帐户数据

Question

dealsgoround.com and citypockets.com access user accounts and their daily deals from livingsocial.com and other daily deal websites. These websites ask users to give their credentials to sign them in to respective websites and then import account details. LivingSocial and other deal website don't provide API access to user accounts.

I also want such access to other websites and import data from there. But I am unable to find out the backend process by which dealsgoround.com and citypockets.com import data from Groupon and LivingSocial etc.

By now I am able to find out that web crawlers/spiders can be used to scrap data from web pages. But I am not sure web crawlers can be useful in a scenario where we need user sign in and the page urls are encrypted or at least are dynamically generated urls.

Please help me and suggest me a way to do this. If crawlers are the solution to this please provide links to some web crawler APIs that I can use in my .net application.

Thanks

Atif

Answer 1

The ability to sign in using alternate credential such as Facebook or Google ID is called Open Auth or OAuth (pronounced Oath).

Incidentally you should not import the data without the users express permission and be vigilent of security issues.

There is a lot of documentation available and a lot of it rather heavy. The best place to start is to read specification RFC 5849

The OAuth website is also useful http://oauth.net

Answer 2

I was able to find the answer... Yes web crawler is the solution in this scenario.

We can use PHP, ASP.net or any other server side scripting language to send an httppostrequest with post parameters (user_name/password). This will authenticate (sign in) the user. After sign-in we can read the contents of any url with user information who has been signed-in.

Note: In my case this isn't unauthorized access to user accounts as users provide their credentials themselves to import their data from deal websites as in dealsgoround.com and citypocket.com