如何使用Perl访问JavaScript驱动的网页的内容？

Question

I was trying to make a little app with Perl to fetch summoner names of League of Legends from LolKing .

The HTML code has lines like

<tr data-summonername="MatLife TriHard" class="lb_row_rank_4">

so I was just going with something like

use strict;
use warnings;

use LWP::Simple;
use HTML::Parser;

my $find_links = HTML::Parser->new(
  start_h => [
    sub {
      my ($tag, $attr) = @_;
      if ($tag eq 'tr' and exists $attr->{'data-summonername'}) {
        print "$attr->{'data-summonername'}\n";
      }
    },
    "tag, attr"
  ]
);

my $html = get('http://www.lolking.net/leaderboards/#/na/1') or die 'nope';

$find_links->parse($html);

but this give me nothing. Even with attr=class , it give me nothing. I can't fetch the tr element's class for some reason.

Using $attr->{data-summonername} without the single quotes gave me some errors, due to the hyphen I suppose. If I fetch $attr->{href} it works just fine.

Can someone help me out?

Answer 1

The problem is that the HTML for that page is mostly built by your browser using JavaScript after the page has been downloaded. Using LWP::Simple::get will just retrieve the skeleton HTML and the JavaScript code. You can see that if you print $html instead of parsing it.

The usual solution is to use WWW::Mechanize::Firefox which gets an installed Firefox to download and build the page which you can then query. It's a lot more complex than a simple get though, as you have to install Firefox if you don't already have it, as well as the Mozilla MozRepl addon which enables remote control. Even then you may still get problems with accessing the contents of the page before the browser has finished building it, so it's not for the faint of heart.

---

Update

For your interest, here is a solution using WWW::Mechanize::Firefox .

use strict;
use warnings;

use WWW::Mechanize::Firefox;
use HTML::TreeBuilder::XPath;

my $url = 'http://www.lolking.net/leaderboards/#/na/1';

my $mech = WWW::Mechanize::Firefox->new;
my $resp = $mech->get($url);
die $resp->status_line unless $resp->is_success;

my $tree = HTML::TreeBuilder::XPath->new_from_content($resp->content);

for my $node ( $tree->findnodes('//tr[starts-with(@class, "lb_row_rank")]') ) {
  printf "Rank %2d: %s\n",
      $node->attr('class') =~ /(\d+)/,
      $node->attr('data-summonername');
}

output

Rank  1: Doublelift
Rank  2: F5 Veritas
Rank  3: Life Love Live 
Rank  4: MatLife TriHard
Rank  5: TDK Kyle
Rank  6: Liquid FeniX
Rank  7: Liquid Inori TV
Rank  8: dawoofsclaw
Rank  9: who is he
Rank 10: Ohhhq