How can I “drill down” into a website using Perl's WWW::Mechanize

Question

I have used the WWW::Mechanize Perl module on a number of projects and it's helped me out a lot.

I am trying to use it on a different site and I can't "drill down" into the content of the site.

The site is https://customer.bookingbug.com/?client=hantsrecyclingcentres#/services

I have tried figure out what the URL would be to get content shown in the resulting HTML, such as bb.d570283b87c834518ba9.css, bb.d570283b87c834518ba9.js and version.js

I tried to copy the resulting HTML into this posting, but used all sorts of quote and code sample combinations and it wouldn't display properly.

Does anyone have any idea how I "navigate" this site using this Perl module please?

Answer 1

WWW::Mechanize is a web client with some HTML parsing capabilities. But as you clearly noticed, the information you want is not in the HTML document you requested. Either download the correct document (whatever that might be), or do what the browser does and execute the JavaScript. This would require a JavaScript engine. The simplest way to achieve that is to remote-control a web browser (eg using Selenium::Chrome ).