AJAX使用Perl WWW：机械化单击按钮

Question

I am carrying out a project for a client and he needs to be able to sweep the table of contents on a particular page. I have modified his existing code to run a loop as there are now multiple pages to extract content from. One of the pages I'm trying to sweep from: https://marriage.ag.gov.au/marriagecelebrants/civil

You can see there's 162 pages which looks to be running on AJAX to load the next lot of content. The existing code would click based on input name attribute:

ctl00$MainContent$gridCelebrants$ctl00$ctl02$ctl00$ctl04 so far all my code does is essentially refresh the page and sweeps the same content 162 times.

This is a current snippet:

use warnings;
use WWW::Mechanize;
use Data::Dumper;
use HTML::TableExtract;
use Spreadsheet::WriteExcel;

#header();
# create max page array to handle civil and other page.
# number indicates how many times to click through
# first item in array is https://marriage.ag.gov.au/marriagecelebrants/civil
# second item is         https://marriage.ag.gov.au/marriagecelebrants/other
my @max_page_array = qw(
    162
    11
);

# create URL array for the 2 pages to scrape
my @url_array = qw(
    https://marriage.ag.gov.au/marriagecelebrants/civil
    https://marriage.ag.gov.au/marriagecelebrants/other
);
# get size of array
my $url_array_size = scalar @url_array;

# declare vars
my $n = 0;
my $i = 0;
# time to loop through the url's
while( $i < $url_array_size){
    open (raw, ">output-dev-$i.txt");
    close(raw);
    $n = 0;
    my $mech = WWW::Mechanize->new(autocheck => 1);
    $mech->get( $url_array[$i] );

    open (raw, ">>output-dev-$i.txt");
    while($n < $max_page_array[$i]){
        my $c = $mech->content;
        my $te = HTML::TableExtract->new(br_translate => 1,keep_html => 0);
        $te->parse($c);
        foreach my $ts ($te->tables) {
            foreach my $row ($ts->rows) {
                print raw join(',', @$row);
            }
        }

       #this was existing code
       #$mech->click( "ctl00\$MainContent\$gridCelebrants\$ctl00\$ctl02\$ctl00\$ctl04" );

       #tried multiple variations based on documentation and got nowhere
       $mech->click_button( 'ctl00$MainContent$gridCelebrants$ctl00$ctl02$ctl00$ctl04' );
       $n++;
    }
    close raw;
    $i++;
} # while loop - url array size 

My question is, when you click next, how can I get my perl script to to load the next page and sweep next set of data?

Answer 1

My question is, when you click next, how can I get my perl script to to load the next page and sweep next set of data?

WWW::Mechanize does not support JavaScript, as per the FAQ . It provides a list of alternatives that do, see also WWW::Mechanize::PhantomJS .