[英]Handling GET errors in WWW::Mechanize
I'm using a script that scrapes data from a website using WWW::Mechanize and it's all working great, except for the website itself. 我正在使用一个脚本,该脚本使用WWW :: Mechanize从网站上抓取数据,并且除了网站本身之外,其他所有功能都运行良好。 Sometimes it just doesn't respond for a short moment and for a given my $url = 'http://www.somesite.com/more/url/text'
I will have this error on $mech->get($url)
: 有时它只是暂时没有响应,并且对于给定的my $url = 'http://www.somesite.com/more/url/text'
我在$mech->get($url)
:
Error GETing http://www.somesite.com/more/url/text: Can't connect to www.somesite.com:443 at ./trackSomesite.pl line 34.
This error is something that occurs once in a while with no recognizable pattern and from my experience with the website I'm dealing with, it's because of server instabilities. 此错误有时会以无法识别的模式偶尔发生,并且根据我与我正在处理的网站的经验,这是由于服务器不稳定。
I want to be able to know specifically that this error occurred and not other errors like Too many requests
. 我希望能够明确知道发生此错误,而不是其他错误(例如Too many requests
。 My question is how can I get my script to handle this error and not die? 我的问题是如何使我的脚本处理该错误而不死?
Either wrap your $mech->get(...)
requests in an eval block or use autocheck => 0
, then check the $mech->status
code and/or $mech->status_line
to decide what to do. 将$mech->get(...)
请求包装在一个eval块中,或者使用autocheck => 0
,然后检查$mech->status
代码和/或$mech->status_line
来决定要做什么。
Here is an example: 这是一个例子:
#!/usr/bin/env perl
use WWW::Mechanize;
use constant RETRY_MAX => 5;
my $url = 'http://www.xxsomesite.com/more/url/text'; # Cannot connect
my $mech = WWW::Mechanize->new( autocheck => 0 );
my $content = fetch($url);
sub fetch {
my ($url) = @_;
for my $retry (0 .. RETRY_MAX-1) {
my $message = "Attempting to fetch [ $url ]";
$message .= $retry ? " - retry $retry\n" : "\n";
warn $message;
my $response = $mech->get($url);
return $response->content() if $response->is_success();
my $status = $response->status;
warn "status = $status\n";
if ($response->status_line =~ /Can['']t connect/) {
$retry++;
warn "cannot connect...will retry after $retry seconds\n";
sleep $retry;
} elsif ($status == 429) {
warn "too many requests...ignoring\n";
return undef;
} else {
warn "something else...\n";
return undef;
}
}
warn "giving up...\n";
return undef;
}
Output 产量
Attempting to fetch [ http://www.xxsomesite.com/more/url/text ]
status = 500
cannot connect...will retry after 1 seconds
Attempting to fetch [ http://www.xxsomesite.com/more/url/text ] - retry 1
status = 500
cannot connect...will retry after 2 seconds
Attempting to fetch [ http://www.xxsomesite.com/more/url/text ] - retry 2
status = 500
cannot connect...will retry after 3 seconds
Attempting to fetch [ http://www.xxsomesite.com/more/url/text ] - retry 3
status = 500
cannot connect...will retry after 4 seconds
Attempting to fetch [ http://www.xxsomesite.com/more/url/text ] - retry 4
status = 500
cannot connect...will retry after 5 seconds
giving up...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.