How to convert a nested for-loop to a recursion?

Question

I get some data from a Memcached, but the asynchrony of nodejs completely eludes me. I want to put all the results into an object.

This is what I would do normally:

for( x = startX; x <= endX; x++ )
{
  for( y = startY; y <= endY; y++ )
  {
    oData[ x + '_' + y ] = Db.get( x + '_' + y );
  } 
}

But I can't figure out how

The Db.get() function wants a key and a callback ( function(error, result) {} )

This would just increment the x...

var onGet = function (error, result)
{
  x++;
  if(!error && result !== null) 
  {
    oData[ x + '_' + y ] = result
  }
};

Db.get(x + '_' + y, onGet);

Answer 1

This is not a recursion problem, it's an "async problem." Your issue is that NodeJS access memcached asynchronously and your procedural style code does not. So, you need to think about the problem differently.

function getMemcacheData(key, data)
{
    DB.get(key, function(err, result)
    {
        if (err) return false;

        data[ key ] = result;
    })
}

var oData = {};

for (var x = startX; x <= endX; x++)
{
    for (var y = startY; y <= endY; y++)
    {
        var key = x + "_" + y;
        getMemcacheData(key, oData);
    }
}

That will work, but - you have another problem then. You have no way of knowing when your MemcacheD data has been loaded into oData , you just have to sit and wait and guess. There are ways to deal with this; but my favorite way is to use a library named async .

With async, you could do this:

var syncStack = [],
    oData = {};

for (var x = startX; x <= endX; x++)
{
    for (var y = startY; y <= endY; y++)
    {
        (function(key)
        {
            syncStack.push(function(callback)
            {
                DB.get(key, function(err, result)
                {
                    /** If you don't care about data not being loaded 
                    you do this: */
                    if (!err)
                    {               
                        data[ key ] = result;
                    }

                    callback();

                    /** If you do care, and you need to terminate if 
                    you don't get all your data, do this: */
                    if (err)
                    {
                        callback(err);
                        return false;
                    }

                    data[ key ] = result;

                    callback();
                })                    
            });
        })(x + "_" + y);  
    }
}

async.parallel(syncStack, function(error)
{
    //this is where you know that all of your memcached keys have been fetched.
    //do whatever you want here.

    //if you chose to use the 2nd method in the fetch call, which does
    //"callback(error)" error here will be whatever you passed as that
    //err argument
});

What this code is actually doing is creating an array of functions, each of which calls the Db.get method for a specific key, and adds the result to the oData variable.

After the array of functions is created, we use the async library's parallel method, which takes an array of functions and calls them all in parallel. This means that your code will send a bunch of requests off to memcached to get your data, all at once. When each function is done, it calls the callback function, which tells the async lib that the request finished. When all of them are finished, async calls the callback closure you supplied as the 2nd argument in the call to the parallel method. When that method is called, you either know A. something went wrong, and you have the error to recover from it or B. all requests are finished, and you may or may not have all the data you requested (expired or stale keys requested, or whatever). From there, you can do whatever you want, knowing you have finished asking for all the keys you needed.

I hope this helps.

Answer 2

parallel get:

function loadPoints(cb)
{
  var oData = {};
  for( x = startX; x <= endX; x++ )
  {
    for( y = startY; y <= endY; y++ )
    {
       var key = x + '_' + y;     
       num_points++;
       Db.get(key, function(err, val) {
         if (err)
           return cb(err);

         oData[key] = val;
         num_points--;
         if (num_points === 0)
            cb(null, oData);         
       });
    } 
  }
}

sequential get:

function loadPoints(cb)
{
   var oData = {};

   function loadPoint(x, y)
   {
      var key = x + '_' + y;
      Db.get(key, function(err, val) {
         if (err)
           return cb(err);
         oData[key] = val;
         if (x + 1 < endX)
            return loadPoint(x + 1, y);
         else if (y + 1 < endY)
            return loadPoint(startX, y+1);
         else
            cb(null, oData);
      });
   }
   loadPoint(startX, startY);
}