使用nodejs复制大量文件？

Question

I have to copy a large amount of files (like 25000) asynchronously. I'm using this library: https://github.com/stephenmathieson/node-cp .

This is my code:

         for(var i = 0; i < 25000; i++){
            cp(origin[i], dest[i], function(err){
              console.log("successfully copied")
            })
         }

It completes the loop but it doens't copy every item. The "Successfully copied" is called between 6000 and 8000 times. After that it doesn't copy anymore. It has something to do with the memory or a limit for async tasks?

Any help would be appreciated!

Answer 1

The copy function takes a callback which is usually a good clue that it's asynchronous. That means that the for loop will continue to run even though the copy hasn't completed, meaning you just queued up 25,000 copy operations!

There's a few ways to solve this but among the most common is using the async module.

var async = require('async');
async.forEachOf(origin, function (file, i, callback) {
    cp(file, dest[i], function(err){
        callback();
    });
})

This won't proceed to the next iteration of the loop until callback is called.

Answer 2

you can copy async with this,

var fs = require('fs-extra')

fs.copy('/tmp/myfile', '/tmp/mynewfile', function (err) {
  if (err) return console.error(err)
  console.log("success!")
}) // copies file 

fs.copy('/tmp/mydir', '/tmp/mynewdir', function (err) {
  if (err) return console.error(err)
  console.log('success!')
}) // copies directory, even if it has subdirectories or file

info => https://www.npmjs.com/package/fs-extra