Converting Javascript 2d arrays to ArrayBuffer

Question

I'm trying to use Web Workers to process large volumes of data, and when passing data back to the main thread for display, I would like to use a transferable object to reduce the impact on the UI thread.

The procedure currently results in a multi dimensional array that can also contain objects. For instance:

[{foo: [{bar: "Alice",
         car: 23,
         dab: [2, 3, 5]}],
  faa: [{moo: {a: [2,3], b: [4,5]} },
        {moo: {a: [6,7], b: [8,9]} }]},
 {foo: [{bar: "John",
         car: 33,
         dab: [6, 7, 1]}],
  faa: [{moo: {a: [5,5], b: [9,2]} },
        {moo: {a: [7,7], b: [4,2]} }]},
 ...]

I have seen this string conversion post, but again, I can't see how to directly apply this to my array structure: Converting between strings and ArrayBuffers

Appreciate the help!

Answer 1

Lot's of people have problem understanding this. So let me give you a image of your options and what they do:

(a) Using plain postMessage with your data

var object = { ... };
worker.postMessage(object);

1. [Main thread] Creates structured clone object
2. [Main thread] Recursively copies data from object to structured clone
3. [Main thread] Posts the object to the [Worker]
4. [Worker] Create new object from structured clone.
5. [Worker] Dispatch new message with object as parameter

Note that creating and parsing structured clone is done by optimized native code.

(b) Converting data to transferable

var object = { ... };
var binary = CreateTypedArrayFromObject(object);
worker.postMessage(binary.buffer, [binary.buffer]);

1. [Main thread] Runs slow javascript code to convert object to TypedArray
2. [Main thread] Which involves either calculating object size first, or creating many typed arrays and concatenating them
3. [Main thread] Moves the ArrayBuffer of the TypedArray to the [Worker]
4. [Worker] Receive ArrayBuffer
5. [Worker] Dispatch new message with object as parameter
6. [Worker] Run javascript code to create new object, discarding received array buffer

---

What I'm pointing out is that you wanted to avoid copy, but you're still making a copy, only this time it's not native but javascript copy. If you want to optimize, you have to design your data structure so that it operates on typed arrays. If it doesn't, just don't even try to use them - you will just add extra overhead to your code.