使用 Node.js 抓取网页

Question

I was wondering if someone can give me an example of how to web scrape multiple pages with Node.js? I am finding examples online but I want to scrape a search result webpage. Then if there is a "next" button to move to the next page and view more results, I want to move to the next page and scrape it as well.

Has anyone done something similar to this?

Thanks!

Answer 1

I managed something like this to work using nightmare.js . It let's you click('#someElement') and wait('#someElement') , getting the page content between these actions using evaluate . It must be in websites that let you do that , note that you may need a while using the nightmare.exists or a for using the page count, for that you may need to use a query selector that can get all specified elements, like document.querySelectorAll('.nextPageElement').length (using the querySelectorAll ) to get that count. Just try to keep using variable lifting when needed and don't fall in a callback hell then nightmare.js will do the job.