Massive memory/RAM usage with Chokidar Node.js package from pre-existing files in directory being watched unnecessarily

Question

I'm using the Node.js package "chokidar" for this use case:

• I'm just watching a single directory on Linux, not recursively
• I only need it to watch for add events, for when files are atomically moved into the watched directory (they're moved from another directory on the same filesystem once all changes are done)
• I don't need it to look at pre-existing files in the dir at all, just watch for new ones since the app started

Problem:

• When there is a large number of pre-existing files (like 80k) in the directory when I start my node.js app, Chokidar seems to put a watch on all of them, even though I don't care about watching existing files at all
• Initially this was hitting the kernel watch limit, but I've already solved that with:

...

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

• The remaining issue is that all these pre-existing files are being watched, which uses a massive amount of RAM for no reason, as I don't need it watching those files at all.
• Is there any way to have Chokidar just entirely ignore all these pre-existing files, and do nothing but watch this single directory for add events for new files only?
• I know about increasing Node's RAM limits (which didn't work in this case anyway), but I feel like it shouldn't be using all this RAM in the first place, and I want it to be efficient on a small VPS. I'd like to solve the RAM usage issue rather than just give it more RAM than it should need in the first place.

I'm using this code:

chokidar.watch("/my-watched-dir", {
    ignoreInitial: true,
})
.on('add', (filepath) => {...}

I've also tried setting the depth option to 0 and 1 too.

The memory usage climbs very high as soon as the app starts (even before the 1st new file appears triggering the add event for the first time).

And there's no problem when the number of pre-existing files is smaller, so it's not an issue related to through-put of new files after the app starts.

Answer 1

As far as I know, libraries like chokidar on Linux platforms will directly use fs.watch and fs.watchFile provided by Node.js.

To be cross-platform, these two APIs always listen for all events related to paths, so the answer is that you can't use chokidar for your purposes.

If you wish to use less memory, either poll manually or use a native Linux module that has direct access to inotify.