Check if an url is blocked by robots.txt using Perl

Question

Can anybody tell me sample code to check if an url has been blocked by robots.txt? We can specify full url or directory in the robots.txt. Is there any helper function in Perl?

Answer 1

Check out WWW::RobotRules :

   The following methods are provided:

   $rules = WWW::RobotRules->new($robot_name)
  This is the constructor for WWW::RobotRules objects.  The first
  argument given to new() is the name of the robot.

   $rules->parse($robot_txt_url, $content, $fresh_until)
  The parse() method takes as arguments the URL that was used to
  retrieve the /robots.txt file, and the contents of the file.

   $rules->allowed($uri)
  Returns TRUE if this robot is allowed to retrieve this URL.

Answer 2

WWW::RobotRules is the standard class for parsing robots.txt files and then checking URLs to see if they're blocked.

You may also be interested in LWP::RobotUA , which integrates that into LWP::UserAgent , automatically fetching and checking robots.txt files as needed.

Answer 3

Load the robots.txt file and search for "Disallow:" in the file. Then check if the following pattern (after the Disallow:) is within your URL. If so, the URL is banned by the robots.txt

Example - You find the following line in the robots.txt:

Disallow: /cgi-bin/

Now remove the "Disallow: " and check, if "/cgi-bin/" (the remaining part) is directly after the TLD.

If your URL looks like:

www.stackoverflow.com/cgi-bin/somwhatelse.pl

it is banned.

If your URL looks like:

www.stackoverflow.com/somwhatelse.pl

it is ok. The complete set of rules you'll find on http://www.robotstxt.org/ . This is the way, if you can not install additional modules for any reason.

Better would be to use a module from cpan: There is a great module on cpan that I use to deal with it: LWP::RobotUA . LWP (libwww) is imho the standard for webaccess in perl - and this module is part of it and ensures your behaviour is nice.

Answer 4

WWW::RobotRules skips rules " substring "

User-agent: *
Disallow: *anytext*

url http://example.com/some_anytext.html be passed (not banned)

Answer 5

Hum, you don't seem to have even looked! On the first page of search results , I see various download engines that handle robots.txt automatically for you, and at least one that does exactly what you asked.