如何在Node.js流回调中聚合由异步函数生成的promise？

Question

I have a Node.js Typescript program in which I'm trying to parse large CSV files line by line and do something with those lines asynchronously. More specifically, I need a function that will:

1. Open a CSV file.
2. Parse the next line to an object.
3. (Ideally) Collect a set number of objects for batch processing.
4. Pass the object(s) to an async function for processing (returns a promise).
5. Collect the promises from the processing function.

Some requirements and considerations:

• I need to poll any of these promises for progress.
• Assume these CSV files are large; streaming line by line is necessary.
• I shouldn't block the application while these processing operations are running.
• Returning an array of promises may not be the right approach, especially if I'm trying to read a file with a million lines.
• I need a hook of sorts to cancel or retry a failed operation.

Here's some test code I've gotten working. ObjectStream is a custom Node.js Transform that converts CSV lines to objects.

function parseFileAsync(filePath: string): Promise<any> {
    var doParseFileAsync = (filePath: string) => {
        var streamDeferred = q.defer<Promise<any>[]>();
        var promises: Promise<any>[] = [];
        var propertyNames: string[] = [];

        var stream = fs.createReadStream(filePath, { encoding: "utf8" })
            .pipe(new LineStream({ objectMode: true }))
            .pipe(new ObjectStream({ objectMode: true }));

        stream.on("readable", () => {
            var obj: Object;
            while ((obj = stream.read()) !== null) {
                console.log(`\nRead an object...`);

                var operationDeferred = q.defer<any>();
                operationDeferred.resolve(doSomethingAsync(obj));
                promises.push(operationDeferred.promise);
            }
        });
        stream.on("end", () => {
            streamDeferred.resolve(promises);
        });

        return streamDeferred.promise;
    }

    return doParseFileAsync(filePath)
        .then((result: Promise<any>[]) => {
            return q.all(result);
        });
}
parseFileAsync(filePath)
    .done((result: any[]) => {
        console.log(`\nFinished reading and processing the file:\n\t${result.toString()}`);
    });

The final done call is executed before the stream even starts running, because parseFileAsync immediately fulfills with an empty array; the stream hasn't had a chance to push any promises yet.

After days of searching, I'm still not sure what the correct way to do this is. Node/JavaScript experts: help?

Update

The code has been updated, and my promises are now playing nicely. However, I need a way to hook into the stream and cancel the process if desired. I also need a way to retry any operations that failed.

Answer 1

I was running into some limitations in the program's architecture that wouldn't allow me to pass promises around as freely as I wanted. So instead, rather than kicking off a bunch of promises, I decided to wait until the previous batch finishes before starting on a new one. Here's the approach I took:

1. Separate the stream stuff into its own function that accepts continuation tokens. The return value will contain the data read as well as a continuation token if there's more data to be read:

    function readFile(filepath: string, lines: number, start: any): Promise<any> { ... } 

2. Define a function that will run the retry-able operation. Within the body of this function, retrieve and process a chunk of data from the file. If the result has a continuation token, "recursively" call the operation function again:

    function processFile(filepath: string, next: any): Promise<any> { var chunkSize = 1; return readLines(filepath, chunkSize, next) .then((result) => { // Do something with `result.lines` ... if (result.next) { return parseFile(filepath, result.next); } }); } 

And voila! A long-running operation that operates on chunks and is easy to report progress on.