I was wondering if someone can give me an example of how to web scrape multiple pages with Node.js? I am finding examples online but I want to scrape a search result webpage. Then if there is a "next" button to move to the next page and view more results, I want to move to the next page and scrape it as well.
Has anyone done something similar to this?
Thanks!
I managed something like this to work using nightmare.js . It let's you click('#someElement')
and wait('#someElement')
, getting the page content between these actions using evaluate
. It must be in websites that let you do that , note that you may need a while
using the nightmare.exists
or a for
using the page count, for that you may need to use a query selector that can get all specified elements, like document.querySelectorAll('.nextPageElement').length
(using the querySelectorAll ) to get that count. Just try to keep using variable lifting when needed and don't fall in a callback hell then nightmare.js will do the job.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.