如何解析多个页面？

Question

I have been attempting to parse a sites table data into a json file, which I can do if I do each page one by one, but seeing as there are 415 pages that would take a while.

I have seen and read a lot of StackOverflow questions on this subject but I don't seem able to modify my script so that it;

1. Scrapes each page and extracts the 50 items with item IDS per page
2. Do so in a rate limited way so I don't negatively affect the server
3. The script waits until all requests are done so I can write each item + item id to a JSON file.

I believe you should be able to do this using request-promise and promise.all but I cannot figure it out.

The actual scraping of the data is fine I just cannot make the code, scrape a page, then go to the next URL with a delay or pause inbetween requests. Code below is the closest I have got, but I get the same results multiple times and I cannot slow the request rate down.

Example of the page URLS:

1. http://test.com/itemlist/1
2. http://test.com/itemlist/2

3. http://test.com/itemlist/3 etc (upto 415)

    for (var i = 1; i <= noPages; i++) { urls.push({url: itemURL + i}); console.log(itemURL + i); } Promise.map(urls, function(obj) { return rp(obj).then(function(body) { var $ = cheerio.load(body); //Some calculations again... rows = $('table tbody tr'); $(rows).each(function(index, row) { var children = $(row).children(); var itemName = children.eq(1).text().trim(); var itemID = children.eq(2).text().trim(); var itemObj = { "id" : itemID, "name" : itemName }; itemArray.push(itemObj); }); return itemArray; }); },{concurrency : 1}).then(function(results) { console.log(results); for (var i = 0; i < results.length; i++) { // access the result's body via results[i] //console.log(results[i]); } }, function(err) { // handle all your errors here console.log(err); }); 

Apologies for perhaps misunderstand node.js and its modules, I don't really use the language but I needed to scrape some data and I really don't like python.

Answer 1

since you need requests to be run only one by one Promise.all() would not help. Recursive promise (I'm not sure if it's correct naming) would.

function fetchAllPages(list) {
    if (!list || !list.length) return Promise. resolve(); // trivial exit
    var urlToFetch = list.pop();
    return fetchPage(urlToFetch).
        then(<wrapper that returns Promise will be resolved after delay >).
        then(function() {
            return fetchAllPages(list); // recursion! 
        });
}

This code still lacks error handling. Also I believe it can become much more clear with async/await:

for(let url of urls) {
    await fetchAndProcess(url);
    await <wrapper around setTimeout>;
}

but you need to find /write your own implementation of fetch() and setTimeout() that are async

Answer 2

After input from @skyboyer suggesting using recursive promises I was lead to a GitHub Gist called Sequential execution of Promises using reduce()

Firstly I created my array of URLS

for (var i = 1; i <= noPages; i++) {
    //example urls[0] = "http://test.com/1"
    //example urls[1] = "http://test.com/2"
    urls.push(itemURL + i);
    console.log(itemURL + i);
}

Then

       var sequencePromise = urls.reduce(function(promise, url) {
         return promise.then(function(results) {
        //fetchIDsFromURL async function (it returns a promise in this case) 
         //when the promise resolves I have my page data
         return fetchIDsFromURL(url)
        .then(promiseWithDelay(9000))
        .then(itemArr => {
          results.push(itemArr);
          //calling return inside the .then method will make sure the data you want is passed onto the next
          return results;
        });
    });
}, Promise.resolve([]));



// async
function fetchIDsFromURL(url)
{
  return new Promise(function(resolve, reject){
    request(url, function(err,res, body){
      //console.log(body);
      var $ = cheerio.load(body);
      rows = $('table tbody tr');
      $(rows).each(function(index, row) {
        var children = $(row).children();
        var itemName = children.eq(1).text().trim();
        var itemID = children.eq(2).text().trim();
        var itemObj = {
          "id" : itemID,
          "name" : itemName
        };
        //push the 50 per page scraped items into an array and resolve with 
        //the array to send the data back from the promise
        itemArray.push(itemObj);
      });
      resolve(itemArray);
    });
 });
}

//returns a promise that resolves after the timeout
function promiseWithDelay(ms)
{
  let timeout =  new Promise(function(resolve, reject){
    setTimeout(function()
    {
      clearTimeout(timeout);
      resolve();
    }, ms);
  });

  return timeout;
}

Then finally call .then on the sequence of promises, the only issue I had with this was returning multiple arrays inside results with the same data in each, so since all data is the same in each array I just take the first one which has all my parsed items with IDs in it, then I wrote it to a JSON file.

  sequencePromise.then(function(results){
  var lastResult = results.length;
  console.log(results[0]);
  writeToFile(results[0]);
});