How can I speed up NodeJS array access?

Question

I wrote a parser in NodeJS that reads a large file of test equipment samples (600MB), de-serializes, and then performs a linear fit of each sample to a table of slopes and offsets (about 10 entries each), eg basically this in a loop:

raw_value = buffer.readUInt16LE(buf_offset)
range = buffer.readUInt16LE(buff_offset+2)
value = raw_value * slope[range] + offset[range]

EDIT: Here's the code. It is proprietary so I had rename everything and change the actual floats, but the idea is the same: a packet comes into the read_packet function hundreds of millions of times. The lookup is the slowest part.

const coeff = {
    mode_x: {
        m: [
            0.0011348513653501,
            0.0011348513653501,
            0.0011348513653501,
            0.0011348513653501,
            0.0003508001973386
        ],
        b: [
            0.0011348513653501,
            0.0011348513653501,
            0.0011348513653501,
            0.0011348513653501,
            0.0003508001973386
        ]
    },
    mode_y: {
        m: [
            0.0000150205642057,
            0.0007774173864163,
            0.0001675113162491,
            0.0000014954827065,
            0.0001675113162491,
            0.0001675113162491,
            0,
            null
        ],
        b: [
            0.0000150205642057,
            0.0007774173864163,
            0.0001675113162491,
            0.0000014954827065,
            0.0001675113162491,
            0.0001675113162491,
            0,
            null
        ]
    }
}

function parse_packet (pkt, length) {
    let accrete = 0
    for (let offset = 0; offset < length; offset += 4) {
        sample = pkt.readUInt16LE(offset)
        xval = pkt.readUInt8(offset + 2)
        yval = pkt.readUInt8(offset + 3)
        x = (i_raw + coeff.mode_x.b[xval]) * coeff.mode_x.b[xval]
        y = (v_raw + coeff.mode_y.m[yval]) * coeff.mode_y.b[yval]
        accrete += x + y
    }
    return accrete
}

I tried moving the m and b arrays out into their own prealloc'd arrays, eg

mode_xm = new Float32Array(50)
coeff.mode_x.m.forEach((x,i) => { mode_xm[i] = x }

But the above made no difference.

If I simply put constants in for slope and offset, the code is ~25 times faster.

Pretty shocked that array/object lookup is so slow compared to C (yes, I know I'm comparing C to JavaScript, bear with my colossal ignorance).

Are there any faster numerical arrays out there? Nothing in npm that I could find. Seems like a plain-old float array with a direct C linkage would be useful.

Thanks, PT

Answer 1

This seems to be a known issue since arrays are objects and the indices are essentially keys in the object code.

Wrong. All engines, including V8 do not use "objects" (meaning: a list of key value pairs) to represent arrays, but use some specialized datastructures depending on the way you use the array.

If I simply put constants in for slope and offset, the code is ~25 times faster.

Right. Constants can be inlined. No calculation is faster than a calculation.

Pretty shocked that array/object lookup is so slow.

How slow? . Thats quite a vague claim.