WWW::Mechanize::Firefox in a high-traffic website

Question

I have a website where I would like to the user to enter a search term and then scrape two other websites and show the user some parsed results.

Since both websites use a lot of JavaScript to return data, I thought of using WWW::Mechanize::Firefox .

Would it be possible to run several simultaneous instances of a script that would use WWW::Mechanize::Firefox ?

Answer 1

Yes, it would be possible to do what you describe; but the internet is not a database.

Have you asked for permission from the owners of the sites that you are hoping to use? I would certainly not agree to an unbounded use of my bandwidth and the contents of my site by a third party.

It would be much more responsible (and easier to implement) a system whereby, by agreement, you gather the information that you need, say, once a day, and store that in a database. Thereafter you can display the information that your own users ask for directly from your own resources.