如何在数组的每个项目上正确地使异步列表异步任务

Question

I need to perform several task in a synchronous way. Extract some files from a folder and its subfolders into an array of items and then make a temp copy of all of them. Those two task I managed to do them synchronously.

Then I need to perform 7 task on every of the item of the array, and I need to wait for the 7 task to be finished for one before passing to the next one.

But every attempt I made ended by doing tasks in this order: Exemple for 3 files to process :

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 6 7 6 7 6 7. And I would like it to be : 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 The lasts two tasks are stacked up and only triggered when the 5 firsts tasks have ne made for every files.

Here I how I start the script :

start();

async function start() {
    await listDirToProcess(); // This works
    await makeSafeCopy(); // This works as well
    for (var currentDCF of dirToProcess) {
        Perform the 7 tasks... 
    }
}

I tried with a callback hell(every thing in the callback of the previous task) I tried without the start() function I tried with more or less async/await keys words I also tried with an asyncForEach function I found in SO instead of the current for-of loop :

async function asyncForEach(array, callback) {
    for (let index = 0; index < array.length; index++) {
        await callback(array[index], index, array);
    }
}

But none of them did the trick.

I'm currently using a for(xx of yy) loop to go through the array.

Here is a summary of those tasks :
Step 1 : Copy files to another folder, i'm doing this like that :

// Usage of the function 
await copyToSomeDir(stuff, (err) => {
    err == 0 ?
        log.info('\t\tCopy to xxx went well') :
        log.info('\t\tCopy to xxx ended with ' + err + " error ");
    });
// Definition of the function
function copyToSomeDir(item, cb) { // Notice i didn't put the async key word
    ...
    cb(i);
}

Step 2 : Wait for an external app to do a job on the Step 1 output folder

// Usage of the function 
await execExternJob();
// Definition of the function
function execExternJob() {
    log.info("Executing Talend job")
    child_process.execSync(cmd_to_exe, (error, stdout, stderr) => {
        log.info("job completed for the " + item.entity);
    })
}

Step 3 : Check the result of the previous extern job :

// Usage of the function 
await checkSuccessfulJob(xxx, (list) => {
    ...
    });
// Definition of the function
function checkSuccessfulJob(xxx, cb) {
    ...
    cb(list);
}

Step 4: Replace some files by others

// Usage of the function 
await OverwriteSucessFile(currentDCF.talendResult.success);
// Definition of the function
function OverwriteSucessFile(list) {
    ...
};

!!! And here is the step that break it all. !!! Step 5 - 6 - 7: If not everything went well during the external job, do stuff Else Zip a folder and upload it to a web service(a CMS actually).

if (fault) {
   fault.forEach((faultyFile) => {
       ... // Not important
   })
} else {
//Usage of the function 
    await prepareZipFolder(xxx, yyy, (path) => { // Step 5
        getUrlFromFile(AlfrescoUrl, (url) => { // Step 6
            uploadToCMS(url, path (ok) => { // Step 7
                        log...
                    })
                });
            });
}

Here is the definition of what I think is the problematic function :

async function prepareZipFolder(zipFileName, rootZipFolder, callback) {
    var zipFile = fs.createWriteStream(path.join(rootZipFolder, zipFileName) + ".zip");
    // >>>>>>>>>>>   Here is where it pass to the next itteration instead of waiting the end of the archiving. I think !     <<<<<<<<<<<
    var archive = archiver('zip', {
        zlib: { level: 9 } // Sets the compression level.
    });
    zipFile.on('close', () => {
        callback(path.join(rootZipFolder, zipFileName) + ".zip")
    })
    zipFile.on('open', (fd) => {
        archive.on('error', function (err) { log.info(err) });
        archive.pipe(zipFile);
        archive.glob("**", { cwd: path.join(rootZipFolder, zipFileName) }, { prefix: zipFileName });
        archive.finalize();
    })
}

The problem is, when the .zip fil is created, the loop pass to the next item instead of waiting for the archiving to be over.

Sorry for the long question, can someone help ?

Answer 1

If I understand correctly, you constantly mix promisified functions with callback functions (also some strange things happen like child_process.execSync(cmd_to_exe, (error, stdout, stderr) => {... — Sync functions do not use callbacks, but this can be a typo).

You need to stick with one approach. If some API does not provide functions that return Promise, these functions will not become Promise-compatible if you just add await before them. You need to find another API or to promisify your wrapper functions by yourself.

For example, your step 2:

await execExternJob();

function execExternJob() {
    log.info("Executing Talend job")
    child_process.execSync(cmd_to_exe, (error, stdout, stderr) => {
        log.info("job completed for the " + item.entity);
    })
}

can be rewritten as:

await execExternJob();

function execExternJob() {
  log.info("Executing Talend job");

  return new Promise((resolve, reject) => {
    child_process.exec(cmd_to_exe, (error, stdout, stderr) => {
      if (error) {
        reject(error);
      } else {
        log.info("job completed for the " + item.entity);
        resolve();
      }
    });
  });
}

And so on.