如何使用cheerio和节点中的请求写入文件？

Question

I am using cheerio and request in node to write a simple scraper, much like this link:

https://www.digitalocean.com/community/tutorials/how-to-use-node-js-request-and-cheerio-to-set-up-simple-web-scraping

my file differs slightly, with the URLs to be scraped in this form: http://foo.com/2014/12/ , http://foo.com/2014/11/ etc. Here it is:

var fs = require('fs');
var request = require('request');
var cheerio = require('cheerio');


var year = ["2014", "2013", "2012", "2011", "2010"];
var yearLength = year.length;

var month = ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'];
var monthLength = month.length;

var postDate = '';
var singlePost = {};

var posts = {
    colors: {
        tastes: []
    }
};

var j, k = 0;

for (j=0; j < yearLength; j++) {
    for (var k=0; k < monthLength; k++) {    

    var fooUrl = 'http://foo.com/' + year[j] + '/' + month[k];         
    var localUrl = './' + year[j] + '/' + month[k] + '/texts.json';

    console.log(JSON.stringify(localUrl));

    request(fooUrl, function (error, response, html) {
        if (!error && response.statusCode == 200) {
            var $ = cheerio.load(html);

            $('ul.slashdot').each(function(i, element){
               var postDate = $(this).children().first().text();            
               $(this).children().first().siblings().each(function(i, element){

               var post = $(this).children();

               var postUrl = post.eq(3).attr('href');



            var singlePost = {

                day: postDate,
                url: postUrl,
                year: year[j],
                month: month[k]

            };

            posts.colors.tastes.push(singlePost);

            });


        });

        fs.writeFile(localUrl, JSON.stringify(posts, null, 2));
        }
    });
    }
}

now when i iterate over the two loops i am unable to write to the file. using writeFileSync doesn't help either.

What am I doing wrong? When I log to console it shows up fine as pretty JSON but doesnt actually write to the file.

Further, year[j] and month[k] are undefined. How do I get them to see the scope of those two variables?

Answer 1

You need to ensure that all directories on the path exists and create them if not.

You may want to take a look at the node-fs-extra module. createFile(file, callback) should help: https://www.npmjs.com/package/node-fs-extra