GET html using WWW::Mechanize causes “Forbidden”

Question

I want to get the content of a film of imdb by using WWW::Mechanize. First of all, I have to find a way to find a respective /title/tt* url. When I have, eg, a movie called fight club, I want to visit this link: *ttp://www.imdb.com/find?s=all&q=fight+club For some reason, this fails already. Heres the line that causes an error

$mech->get('http://www.imdb.com/find?s=all&q=fight+club');

error message:

Error GETing http://www.imdb.com/find?s=all&q=fight+club : Forbidden

If I write something like get(http://www.google.com), it works fine. What's the difference when using imdb? Any proposal for an alternative solution?

Answer 1

IMDB probably sniff the User-Agent string and reject WWW::Mechanize requests. The "solution" is to respect their wish to block you from interacting with the site in an automated fashion.

(Or you could read their terms and conditions very, very carefully and then change the user agent string)

Licensing IMDb Content; Consent to Use Robots and Crawlers: If you are interested in receiving our express written permission to use IMDb content for your non-personal (including commercial) use, please visit our Content Licensing section or contact our Licensing Department. We do allow the limited use of robots and crawlers, such as those from certain search engines, with our express written consent. If you are interested in receiving our express written permission to use robots or crawlers on our site, please contact our Licensing Department.

Answer 2

David is right, that's probably what's happening.

But did you know lots of information is available from IMDB via FTP? And that they have a number of tools you can use to get at their information other than scraping?

See http://www.imdb.com/interfaces