How to make an asynchronous self-calling loop non-recursive

Question

I'm writing a function in PHP that loops through an array, and then performs an asynchronous call on it (using a Promise).

The problem is that, the only way I can make this loop happen, is by letting a function call itself asynchronously. I run into the 100-nested functions problem really quick, and I would basically like to change it to not recur.

function myloop($data, $index = 0) {

    if (!isset($data[$index])) {
        return;
    }

    $currentItem = $data[$index];
    $currentItem()->then(function() use ($data, $index) {
       myloop($data, $index + 1);   
    });

}

For those that want to answer this from a practical perspective (eg: rewrite to not be asynchronous), I'm experimenting with functional and asynchronous patterns and I want to know if it is possible to do this with PHP.

Answer 1

I've written a possible solution in pseudo-code. The idea is to limit the number of items running asynchronous at once by using a database queue. myloop() is no longer directly recursive, instead being called whenever an item finishes running. In the sample data, I've limited it to 4 items concurrently (arbitrary value). Basically, it's still recursively calling itself, but in a roundabout way, avoiding the situation you mentioned of many nested called.

Execution Flow:

myloop() ---> queue
^              v
|              |
'<-processor <-'  

---

<?php    
//----------
// database

//table:  config
//columns: setting, value
//items:    ACTIVE_COUNT, 0
//          ITEM_CONCURRENT_MAX, 4
//table: queue
//columns: id, item, data, index, pid, status(waiting, running, finished), locked
//  --- end pseudo-schema ---

<?php    
// ---------------
// itemloop.php
// ---------------

//sends an item and associated data produced by myloop() into a database queue,
//to be processed (run asynchronous, but limited to how many can run at once) 
function send_item_to_processor($item, $data, $index, $counter) { 
   //INSERT $item to a queue table, along with $data, $index (if needed), $counter, locked = 0
   //status == waiting
}

//original code, slightly modified to remove direct recursion and implement 
//the queue.
function myloop($data, $index = 0, $counter = 0) {

    if (!isset($data[$index])) {
        return;
    }

    $currentItem = $data[$index];
    $currentItem()->then(function() use ($data, $index) {
       //instead of directly calling `myloop()`, push item to
       //database and let the processor worry about it. see below.
       //*if you wanted currentItem to call a specific function after finishing,
       //you could create an array of numbered functions and pass the function
       //number along with the other data.* 
       send_item_to_processor($currentItem, $data, $index + 1, $counter + 1);
    });

}


// ---------------
// processor.php
// ---------------

//handles the actual running of items.  looks for a "waiting" item and 
//executes it, updating various statuses along the way. 
//*called from `process_queue()`*
function process_new_items() { 
   //select ACTIVE_COUNT, ITEM_CONCURRENT_MAX
   //ITEM_COUNT = total records in the queue. this is done to
   //short-circuit the execution of `process_queue()` whenever possible
   //(which is called frequently).
   if (ITEM_COUNT == 0 || $ACTIVE_COUNT >= $ITEM_CONCURRENT_MAX) 
     return FALSE;
   //select item from queue where status = waiting AND locked = 0 limit 1;
   //update item set status = running, pid = programPID
   //update config ACTIVE_COUNT = +1
   //**** asynchronous run item here ****//
   return TRUE;
}  

//main processor for the queue.  first processes new/waiting items
//if it can (if too many items aren't already running), then processes
//dead/completed items.  Upon an item.status == finished, `myloop()` is
//called from this function.  Still technically a recursive call, but 
//avoids out-of-control situations due to the asynchronous nature.    
//this function could be called on a timer of some sort, such as a cronjob
function process_queue() {
  if (!process_new_items()) 
    return FALSE;  //too many instances running, no need to process

  //check queue table for items with status == finished or is_pid_valid(pid) == FALSE
  $numComplete = count($rows);
  //update all rows to locked = 1, in case process_queue() gets called again before
  //we finish, resulting in an item potentially being processed as dead twice. 
  foreach($rows as $item) {
    if (is_invalid(pid) || $status == finished)  { 
       //and here is the call back to myloop(), avoiding a strictly recursive 
       //function call.  
       //*Not sure what to do with `$item` here -- might be passed back to `myloop()`?.*
       //delete item(s) from queue
       myloop(data, index, counter - 1);
       //decrease config.ACTIVE_COUNT by $numComplete
    }
  }
}