使用npm cheerio和and请求动态网页抓取

Question

I'm trying to scrape data from a site which has a base url and then dynamic routes. This particular site simply uses numbers, so I have this code to get the data:

 for (var i = 1; i <= total; i++) { var temp = base_url + i; var result = ""; request(temp, function(error, response, body) { var $ = cheerio.load(body); var address_string = 'http://maps.google.com/?q=' + $('title').text(); //firebase database.ref('events/' + i).set({ "address": address_string }); }); } 

However, the above code doesn't work, and doesn't add anything to the database. Does anyone know what's wrong?

Answer 1

I'm not sure about the reason, but one thing that will behave strangely in the code you wrote is that the variable i is not bound to the callback scope of the request, and the for loop will finish before any callback is called.

If this is the problem, there should only be one db entry for i === total.

This can be solved by doing an Array.forEach instead.