NodeJS stream 返回不完整的响应

Question

I am using lento (Streaming Node.js client for Presto, the "Distributed SQL Query Engine for Big Data") for querying database. Lento's createRowStream takes sql query as a string or Buffer and returns a readable stream that yields rows.

Before returning the resulted row stream, I need to do some pre-processing on the result ( stream.pipe() does that for me) and convert it to CSV format ( csvStringify does that for me)

Once the stream ends, I resolve the promise with resolve() callback and also log the number of rows streamed. But, compared to the number of rows streamed, the actual rows returned are less. For example, if the log says count is 10000 (variable rowsCnt ) the number of rows returned would be close to 6000.

What could be the cause for the inconsistency in number of rows returned?

Please check below imports and code snippet:

import csvStringify from 'csv-stringify';
import {Request, Response} from 'express';
import lento from 'lento';
import streamTransform from 'stream-transform';

async getCSVRows(res: Response, sqlQueries: sqlDto[]): Promise<void> {
    const result = [];
    const lentoClient = ... code to create instance of lento client
    for (let index = 0; index < sqlQueries.length; index++) {
        const csvStream = csvStringify({header: index == 0 ? true : false});
        queryResult = executeQuery(res, sqlQueries[index], lentoClient, csvStream)
        result.push(queryResult)
    }
    await Promise.all(result);
    res.end()
}

async executeQuery(
    res: Response, 
    sqlQuery: sqlDto, 
    lentoClient: any, 
    csvStream: csvStringify.Stringifier
): Promise<Response> {
    return new Promise(async (resolve) => {
        const rowsStream = lentoClient.createRowStream(sqlQuery.query); 
        let rowsCnt = 0;
        rowsStream.on('data', function() {})
        .pipe(
            streamTransform((row: any) => {
                 // process row
                 rowsCnt++;
                 return row;
            }),
        )
        .pipe(csvStream)
        .pipe(res, {end: false});
        rowsStream.on('error', (err: Error) => {
            // log error
            throw err;
        });
        rowsStream.on('end', () => {
             resolve(res);
             console.log('Rows count: ' + rowsCnt);
        });
    });
}

Note: Framework used is NestJS

Answer 1

I think you are ending the responses too early, when there is still processing going on in the pipeline. In your code, when rowsStream emits the 'end' event, doesn't mean its the end of processing. It means the last chunk of query results are in the pipeline, but still processing. You need to end the response after all of the processing is done and the pipeline is empty. It's tricky, because you are piping multiple queries in the same response, so you use the {end: false} option. Because of this, "res" won't emit the end event so you have to detect the end of processing some other way.

Here is my take:

executeQuery(
    res: Response, 
    sqlQuery: sqlDto, 
    lentoClient: any, 
    csvStream: csvStringify.Stringifier
): Promise<Response> {
    return new Promise((resolve) => {
        const rowsStream = lentoClient.createRowStream(sqlQuery.query); 
        let rowsCnt = 0;
        rowsStream
        .pipe(
            streamTransform((row: any) => {
                 // process row
                 rowsCnt++;
                 return row;
            }),
        )
        .pipe(csvStream)
        .on('end', () => {
             nextTick(()=> {
                 resolve();
                 console.log('Rows count: ' + rowsCnt);
             })               
         })
        .pipe(res, {end: false});
        rowsStream.on('error', (err: Error) => {
            // log error
            throw err;
        });
    });
}

You can play around with the promisified pipe version:

    import { pipeline } from 'stream';
    import { promisify } from 'util';
    
    const pipe = promisify(pipeline);
    
    async executeQuery(
      res: Response,
      sqlQuery: sqlDto,
      lentoClient: any,
      csvStream: csvStringify.Stringifier,
    ) {
      let rowsCnt = 0;
      try {
        await pipe(
          lentoClient.createRowStream(sqlQuery.query),
          streamTransform((row: any) => {
            // process row
            rowsCnt++;
            return row;
          }),
          csvStream.pipe(res, {end: false}),          
        );
        console.log('Rows count: ' + rowsCnt);
      } catch (error) {
        // log error
      }
    }