Node.js- share a heavy variable from parent to worker threads

Question

I want to share a variable between parent and worker threads. The variable is heavy (100+ MB - arrays of JSON data) so if I pass it as an argument or re-read from data source, it drastically increases the execution time of the code.

As per official documentation, I understand that:

• parent and worker threads operate with isolated memories (so using global variables is no-go) and
• arguments passed will get fully cloned in the worker memory (cloning increases the execution time).

Any strategy to use a common memory and use its references in both parent and worker threads or any suggestions to enable the sharing is welcome. Please find the model code below I am working with. I am 'piscina' library so far for pool management as of now.

main.js

const piscina = require('piscina'), { resolve } = require('path');
const heavyVar = require('./heavy-file.json');

async () => {
    const argLst = [1, 2, 3];
    const pool = new piscina();
    const option = { filename: resolve(__dirname, 'worker.js') };

    for (let arg of argLst) {
        requests.push(pool.run({ arg, heavyVar }, option));
    }
}

worker.js

async ({ arg, heavyVar }) => {
    // let heavyVar = require('./heavy-file.json');  
    // Reading the same file again or passing argument worker increases execution time severely

    let result = heavyVar.filter(r => r.prop == arg);
    console.log(result);
}

Answer 1

The .postMessage() operation uses so-called transferable objects . When your main thread uses it to send an object to a worker, or vice versa, the object becomes inaccessible in the sending thread.

ArrayBuffers and some other objects are transferred directly, with high performance, via some sort of low-level V8 javascript engine address space voodoo without copying them.