How to avoid memory leak when creating promises recursively?

Question

I have an ionic application which needs to download data (with pagination) and insert it into database recursively (like a pipeline). I'm creating promises with (Angular's) $q service. The problem is that when I call this recursive function, the data is being downloaded and inserted successfully, however memory usage is always increasing, and when the promise chain resolves completely, allocated memory still remains in use.

Here is my recursive function:

// offset:  last downloaded row count
// limit:   row count to download at each page
// numRows: row count to download at all
function dowloadAndInsert(offset, limit, numRows) {
    var deferred = $q.defer();

    // Recursion step: We do not reached at the end of data
    if((offset + limit) <= numRows) {

        // Download the data
        downloadData(offset, limit)
            .then(function(response) {

                // Insert the data
                insertData(response)
                    .then(function(insertTime) {

                        // Recursion step
                        dowloadAndInsert(offset + limit, limit, numRows)
                            .then(function() {
                                deferred.resolve();
                            })
                            .catch(function(reason) {
                                deferred.reject(reason);
                            });
                    })
                    .catch(function(reason) {
                        deferred.reject(reason);
                    });
            })
            .catch(function(reason) {
                deferred.reject(reason);
            });
    }

    // Base case: We reached at the end of data
    else {
        var remainingRows = numRows % limit;        // Means the last limit actually

        // If exists, insert remaining rows
        if(remainingRows !== 0) {

            // Download the last piece of data
            downloadData(offset, remainingRows)
                .then(function(response) {

                    // Insert the last piece of data
                    insertData(response)
                        .then(function(insertTime) {

                            // Base case, successfully downloaded and inserted everything
                            deferred.resolve();
                        })
                        .catch(function(reason) {
                            deferred.reject(reason);
                        });
                })
                .catch(function(reason) {
                    deferred.reject(reason);
                });
        }

        else {
            // Base case, successfully downloaded and inserted everything
            deferred.resolve();
        }
    }

    return deferred.promise;
}

Note: response object coming from downloadData function is a big data, it sometimes contains 100.000 rows with 18 columns. Total memory usage is becoming like 1GB. I'm running my tests on iPad Air 2.

I'm playing with big data in my recursion function, so that my question is a bit different from other recursive memory leak questions.

Thanks.

Answer 1

Your code is working way too hard, promises chain, so when you do the little deferred dance - you could really just chain promises which should resolve the leak as a side effect of the better code:

function dowloadAndInsert(offset, limit, numRows) {
  const start = offset,
        numFetch = ((offset + limit) <= numRows ? limit : numRows % limit;
  if(numFetch === 0) {
     return Promise.resolve(); // we're done;
  }
  return downloadData(start, end).
           then(insertData).
           then(downloadAndInsert.bind(null, offset + numFetch, limit, numRows);
}

And that's the entire code, it says:

• Check how many rows I need to fetch and insert.
• If I don't need to fetch anymore rows - just return an empty promise.
• Otherwise fetch as many as we need - and then fetch the remaining rows.