Why hash in Node.js give different result with same character?

Question

So i was try to hash ¤ character in node js, with this function crypto.createHash('md5').update('¤', 'ascii').digest('hex') give md5 hash

f37c6f3896b2c85fbbd01ae32e47b43f

and using Buffer

crypto.createHash('md5').update(new Buffer('¤', 'ascii').toString()).digest('hex')

give result like this:

9b759040321a408a5c7768b4511287a6

I tried to debug Hash.update() to take a look inside but i can't it seems hard compiled.

Why crypto encoding method is different with Buffer ? what makes it different?

Answer 1

crypto is encoding the same way as buffers do, so let's ignore it for now. Here's a simplification of the issue:

const text = '¤';
const b1 = Buffer.from(text, 'ascii');
const b2 = Buffer.from(b1.toString());

b1 and b2 aren't the same bytes. b1 is [0xa4], which doesn't really make much sense as 0xa4 isn't part of ASCII; Node is using the same code to encode strings as ASCII and Latin-1 here . I don't know if that's for compatibility or performance reasons or what, but it seems like a bad idea, results in values for which Buffer.from(s, 'ascii') is different from Buffer.from(Buffer.from(s, 'ascii').toString('ascii'), 'ascii') , and does not appear to be documented anywhere.

In modern versions of Node, the default encoding is UTF-8, so b1.toString() will try to interpret 0xa4 as UTF-8, fail, and produce a replacement character ( ) instead, encoded as [0xef, 0xbf, 0xbd]. In non-modern versions of Node, it will do an environment-dependent wrong thing instead of a consistent wrong thing.

You can make your operations give the same result by passing a buffer instead of a UTF-8 encoding of a buffer:

crypto.createHash('md5').update(new Buffer('¤', 'ascii')).digest('hex')

(note how .toString() is removed)

but correct code, able to hash any sequence of Unicode codepoints, would use UTF-8 instead.

crypto.createHash('md5').update('¤', 'utf8').digest('hex')
crypto.createHash('md5').update(Buffer.from('¤', 'utf8')).digest('hex')