How can I extract (read + delete) from a textfile in NodeJS?

Question

I'm building a script that reads log files, handles what needs to be handled then writes them to a database

Some caveats :

Some log files have a lot of input, multiple times a second Some log files have few to no input at all

What I try in simple words:

Reading the first line of a file, then deleting this line to go to the next one, while I handle the first line, other lines could be added..

Issues I'm facing

1. When I try reading a file then processing it, then deleting the files, some lines have been added
2. When the app crashes while handling multiple lines at once for any reason, I can't know what lines have been processed.

Tried so far

fs.readdir('logs/', (err, filenames) => {
filenames.forEach((filename) => {
  fs.readFile('logs/'+filename, 'utf-8', (err, content) => {

    //processing all new lines (can take multiple ms)

    //deleting file
    fs.unlink('logs/'+filename)
  });
});

});

Is there not a (native or not) method to 'take' first line(s), or take all lines, from a file at once?

Something similar to what the Array.shift() method does to arrays..

Answer 1

If your log files has been writen as rotate logs. Example: Each hours has each log file, 9AM.log, 10AM.log....When you process the log files, you can skip current file and process another files. ex: now is 10:30 AM o'clock, skip file 10AM.log, solve another files.

Answer 2

Why you are reading the file at once. Instead you can use the node.js streams . https://nodejs.org/api/fs.html#fs_class_fs_readstream

This will read the files and output to console

var fs = require('fs');
var readStream = fs.createReadStream('myfile.txt');
readStream.pipe(process.stdout);

You can also go for the npm package node-tail to read the content of a files while new content written to it. https://github.com/lucagrulla/node-tail