如何在Perl Web爬虫中处理Javascript？

Question

I would like to crawl a website, the problem is, that its full of JavaScript things, such as buttons and such that when they are pressed, they do not change the URL, but the data on the page is changed.

Usually I use LWP / Mechanize etc to crawl sites, but neither support JavaScript. any idea?

Answer 1

The WWW::Scripter module has a JavaScript plugin that may be useful. Can't say I've used it myself, however.

Answer 2

另一种选择可能是Selenium与WWW :: Selenium模块

Answer 3

WWW::Mechanize::Firefox might be of use. that way you can have Firefox handle the complex JavaScript issues and then extract the resultant html.

Answer 4

I would suggest HtmlUnit and Perl wrapper: WWW::HtmlUnit .