Node.js parallel tasks with Napajs

Question

I'm trying to sort array of 10million values. Found new technology from Microsoft - Napa.js that allows use multi-threading with Node.js.

There we use following code:

const napa = require('napajs')
const zone = napa.zone.create('zone', { workers: 4 });
zone.execute((...args) => {
  // here I can take anything I want to execute in a native thread
  // but it is also possible to implement shared memory for all 
  // threads, but I can't find out how
}, [args])

Help me to understand, what to allocate . what is handler and how to use TransferContext if it needed for this task.

Ant at least how to use all this knowledge to create shared memory for my threads ?

Answer 1

An update:

Napa.js (>=0.2.0) now supports transporting JavaScript built-in types such as SharedArrayBuffer without copying. You may find example Parallel Quick Sort for particular this problem.

---

This is an interesting problem. Currently, arguments passing between workers are serialized/deserialized by default in Napa.js. We may want a better solution of sharing array across workers without copying.

I saw the feasibility as following:

1. Assume we use int32 array to hold the 10M values in main JS thread.
2. Create an addon to get the raw pointer of the array via v8::Int32Array::Buffer()::Externalize(). (This let user to manage the memory, we may need some lifecycle management)
3. Pass the raw pointer to another worker.
4. Worker to create an v8::Int32Array from a raw pointer (through a externalized ArrayBuffer).

This pattern is so common that I think it might be a good idea to have it included in Napa.js. I have placed a feature request Introduce Buffer class that can be transferred across workers without copying .

BTW, TransportContext is used to transfer std::shared_ptr from one worker to another worker, in this case I think it's more about how we share array across workers.