How to download image folder from website to local directory using nodejs

Question

I would like to download image folder which contains number of images. I need to download into my local directory. I downloaded one image giving image name. But I could not understand how can I do that for multiple images itself. Here is my code.

var http = require('http');
var fs = require('fs');

var file = fs.createWriteStream("./downloads");
var request = http.get("http://www.salonsce.com/extranet/uploadfiles" + image.png, function(response) {
  response.pipe(file);
});

Thanks in advance.

Answer 1

To download files using curl in Node.js you will need to use Node's child_process module. You have to call curl using child_process's spawn method. for that i use spawn instead of exec for the sake of convenience - spawn returns a stream with data event and doesn't have buffer size issue unlike exec. That doesn't mean exec is inferior to spawn; in fact we will use exec to download files using wget.

// Function to download file using curl
var download_file_curl = function(file_url) {

    // extract the file name
    var file_name = url.parse(file_url).pathname.split('/').pop();
    // create an instance of writable stream
    var file = fs.createWriteStream(DOWNLOAD_DIR + file_name);
    // execute curl using child_process' spawn function
    var curl = spawn('curl', [file_url]);
    // add a 'data' event listener for the spawn instance
    curl.stdout.on('data', function(data) { file.write(data); });
    // add an 'end' event listener to close the writeable stream
    curl.stdout.on('end', function(data) {
        file.end();
        console.log(file_name + ' downloaded to ' + DOWNLOAD_DIR);
    });
    // when the spawn child process exits, check if there were any errors and close the writeable stream
    curl.on('exit', function(code) {
        if (code != 0) {
            console.log('Failed: ' + code);
        }
    });
};

Answer 2

A better way to do this would be to use another tool called glob in parallel. Like,

First install it with

npm install glob

And then,

var glob = require("glob");
var http = require('http');
var fs = require('fs');

var file = fs.createWriteStream("./downloads");

// options is optional
//options = {};
glob('http://www.salonsce.com/extranet/uploadfiles/*', options, function (er, files) {
  //you will get list of files in the directory as an array.
  // now use your previus logic to fetch individual file
  // the name of which can be found by iterating over files array
  // loop over the files array. please implement you looping construct.
  var request = http.get(files[i], function(response) {
      response.pipe(file);
  });

});