抓取需要触发javascript的外部网站

Question

Since phantomjs is abandoned, I would like to know if there is any alternative method. eg chrome-webdriver wouldn't be a good solution as it wouldn't be able to run on a remote host such as heroku.

So, is it somehow possible to scrape an external website that require javascript being triggered first? Note that it should be possible to run it from a nodejs application.

Answer 1

I was getting ready to put together something for you, then I thought better and google'd it. Check out this build script; it seems to answer your question exactly.

https://github.com/stomita/heroku-buildpack-phantomjs

Set up a git branch and pull it locally if you have to, but this should work. Basically, you need to download the binary and then remote in and run "heroku run 'phantomjs'" or "heroku run 'bin/phantomjs'"