NodeJS exec stop after some time without error

Question

I'm using exec from child_process .

The function runs fine but after 4-5minutes, it just stops, without any errors reported, but the script should run for at least 24hours...

Here is the code :

import { exec } from 'child_process';

function searchDirectory(dirPath) {
    let lineBuffer = '';

    const cmd = `find ${dirPath} -type f -name "*.txt" | pv -l -L 10 -q`;
    const findData = exec(cmd);

    findData.on('error', err => log.error(err));
    findData.stdout.on('data', data => {
        lineBuffer += data;
        let lines = lineBuffer.split('\n');
        for (var i = 0; i < lines.length - 1; i++) {
            let filepath = lines[i];

            processfile(filepath);
        }
        lineBuffer = lines[lines.length - 1];
    });
    findData.stdout.on('end', () => console.log('finished finding...'));
}

The pv command slows down the output, I need this since the path where I'm finding is over the network and pretty slow (60mb/s).

When I run the command directly in the terminal it works fine (I didn't wait 24hours but I let it for half hour and it was still running).

The processfile function actually makes an async call with axios to send some data to a server :

    let data = readFileSync(file);
    ...
    axios.post(API_URL, { obj: data }, { proxy: false })
        .then(res => {
            log.info('Successfully saved object : ' + res.data._id);
        })
        .catch(err => {
            log.error(err.response ? err.response.data : err);
        });

What could cause the script to stop? Any ideas?

Thanks

Answer 1

I found the issue, using exec is not recommended for huge outputs since it's using a limited size buffer. Use spawn instead :

The most significant difference between child_process.spawn and child_process.exec is in what they return - spawn returns a stream and exec returns a buffer.

child_process.spawn returns an object with stdout and stderr streams. You can tap on the stdout stream to read data that the child process sends back to Node. stdout being a stream has the "data", "end", and other events that streams have. spawn is best used to when you want the child process to return a large amount of data to Node - image processing, reading binary data etc.

child_process.exec returns the whole buffer output from the child process. By default the buffer size is set at 200k. If the child process returns anything more than that, you program will crash with the error message "Error: maxBuffer exceeded". You can fix that problem by setting a bigger buffer size in the exec options. But you should not do it because exec is not meant for processes that return HUGE buffers to Node. You should use spawn for that. So what do you use exec for? Use it to run programs that return result statuses, instead of data.

from : https://www.hacksparrow.com/difference-between-spawn-and-exec-of-node-js-child_process.html