如何在nodejs中进行大量但未知的REST http调用？

Question

I have an orientdb database. I want to use nodejs with RESTfull calls to create a large number of records. I need to get the @rid of each for some later processing.

My psuedo code is:

for each record
  write.to.db(record) 
  when the async of write.to.db() finishes
     process based on @rid
carryon()

I have landed in serious callback hell from this. The version that was closest used a tail recursion in the .then function to write the next record to the db. However, I couldn't carry on with the rest of the processing.

A final constraint is that I am behind a corporate proxy and cannot use any other packages without going through the network administrator, so using the native nodejs packages is essential.

Any suggestions?

Answer 1

With a completion callback, the general design pattern for this type of problem makes use of a local function for doing each write:

var records = ....;    // array of records to write
var index = 0;

function writeNext(r) {
    write.to.db(r, function(err) {
       if (err) {
          // error handling
       } else {
          ++index;
          if (index < records.length) {
             writeOne(records[index]);
          }
       }
    });
}

writeNext(records[0]);

The key here is that you can't use synchronous iterators like .forEach() because they won't iterate one at a time and wait for completion. Instead, you do your own iteration.

---

If your write function returns a promise, you can use the .reduce() pattern that is common for iterating an array.

var records = ...;    // some array of records to write

records.reduce(function(p, r) {
    return p.then(function() {
       return write.to.db(r);
    });
}, Promsise.resolve()).then(function() {
   // all done here
}, function(err) {
   // error here
});

This solution chains promises together, waiting for each one to resolve before executing the next save.

Answer 2

It's kinda hard to tell which function would be best for your scenario w/o more detail, but I almost always use asyncjs for this kind of thing.

From what you say, one way to do it would be with async.map :

var recordsToCreate = [...];

function functionThatCallsTheApi(record, cb){
   // do the api call, then call cb(null, rid)
}

async.map(recordsToCreate, functionThatCallsTheApi, function(err, results){
   // here, err will be if anything failed in any function
   // results will be an array of the rids
});

You can also check out other ones to enable throttling, which is probablya good idea.