node.js Buffer not empty

Question

When I create lots of Buffer s they aren't always empty:

for (var i = 0; i < 100; i++) {
    console.log((new Buffer(30)).toString('hex'));
}

(Partial) Output:

782668013a0000003b00000035000000b0c17900391100003c0000003d00
e4216801ffffffff000000000100000000000000000000003e0000003f00
40c27900100000000100000000000000000000000000000018c379000000
000000000000000000000000000000000000000000000000000000000000 --> Empty
000000000000000000000000000000000000000000000000000000000000 --> Empty
0000000000000000108269014000000041000000c86f79000cf679000000
6611000080c27900c0c27900040000000100000000000000d0c279000000
00000000000000005c2468014200000043000000cc6f7900002668014400

(Partial) Output (without .toString('hex') and only new Buffer(10) ):

<Buffer 01 00 00 00 58 db 62 00 b4 86>
<Buffer 90 b9 65 00 08 00 00 00 03 00>
<Buffer 10 ba 65 00 04 00 00 00 00 00>
<Buffer 04 00 00 00 00 00 00 00 00 00>
<Buffer 10 00 00 00 00 00 00 00 70 ba>
<Buffer 00 00 00 00 00 00 00 00 00 00> --> Empty
<Buffer 00 00 00 00 00 00 00 00 00 00> --> Empty
<Buffer ff ff ff ff ff ff ff ff 00 00>
<Buffer 00 00 00 00 0f 00 00 00 8c 6f>
<Buffer 80 ba 65 00 00 00 00 00 aa 00>

I'm running node.js v0.10.33 on a 32-bit Windows 7 machine.

• Is this a problem with my specific machine?
• Is this a general problem (aka. bug) of node.js?
• Is this a problem of a special composition (eg only Windows environments)?
• Is this expected and I have to clear the Buffer before I start using it?
• Is this documented?

Update: Better behaviour on v0.11.14 , worse behaviour on v0.8.28

Answer 1

tl;dr

• Not a problem with your specific machine, it's on all machines.
• Not a bug. This behavior exists in all memory allocation libraries.
• Not OS depended. All OS behave this way.
• If you need it initialized, then yes, clear it using Buffer.prototype.fill
• Yes, it's documented in the underlying libraries that nodejs/webkit use: see malloc(3) / stdlib

This is most likely something that's to be expected when working with API that handles memory allocation, such as Buffer.

Buffer is actually using the smalloc module, which you could think of as malloc(3) / free(3) from stdlib.h

The principle behind malloc is that it only allocates/reserves memory for a pointer - in webkit it might look closer to calloc for ExternalArray on objects.

See http://linux.die.net/man/3/malloc

The malloc() function allocates size bytes and returns a pointer to the allocated memory. The memory is not initialized.

Memory allocation/deallocation does not handle memory initialization, because it's more expensive to go through each byte and set it to 0.

Besides 0 is not always the initial value you need for a byte.

And since memory allocation can return a block of memory that has been used before by some other process, it's expected to have data in the newly allocated block.

The general rule is: when allocating memory (such as Buffer does), if you need it initialized use buf.fill(0) ; although this is not always needed since in most scenarios when you need Buffer you already have knowledge of the length of the data, implicitly the contents.

For instance, when you create new Buffer(30) you know your data is 30 bytes long, which means you already have an idea of what the data to be written looks like, so in the end you'll most likely end up with writing each byte, before passing the buffer to something else - therefore no need to prepend an additional initialization loop that will set each byte to 0.

For buffers where you use bytes as flags/states for an object, you could only initialize those.

Eg if your Buffer(10) uses first 2 bytes as flags for some state, and the other 8 bytes for data, only do buffer[0] = buffer[1] = 0 , to set them to initial 0 value, instead of initializing all 10 bytes, and then doing a write on the last 8 anyway.

One last note: If the memory block returned by memory allocation hasn't been used before, then yes, all bytes are set to 0. However, memory handling is something the OS does - using all sorts of optimisations - therefore it's impossible to predict what memory segment you will get and its contents.

Answer 2

In the lastest versions of Node (v5.10.0+) you can use

--zero-fill-buffers

https://nodejs.org/api/buffer.html#buffer_the_zero_fill_buffers_command_line_option