刮擦动态表单WWW :: Mechanize Perl

Question

I am attempting to scrape a form and its fields from a page using the WWW::Mechanize module.

Because the main body of the webpage is created using document.write JS calls, the form methods from this module aren't finding the form I am looking for, and a call to the content method returns the page source. I need to access the resulting HTML from the document.write calls.

Is this possible using the mechanize module and if so how would I go about doing it? If not, are there other Perl modules that would help me? Thanks!

Answer 1

I know that you are for the Perl solution, but you might consider Ruby. I have done multiple web scraping scripts in both Perl and Ruby. I found out that Ruby does a better web scraping job then Perl.

Since you are running on Linux, Ruby should be either already installed or should be a simple installation (assuming you are allowed to do installations on the server).

You can use these threeruby gems for automation:

require 'watir-webdriver'
require 'selenium-webdriver'
require 'headless'

These do a very good job at web scraping.