Why does xxd add chars?

Question

I am trying to reverse an od command from a system where I have no hexdump or base64 tools.

I do this like that (of course, in reality, the encoding takes place at the "small" system, the decoding is done on my workstation, but to test it, I try the whole way in one line first):

echo TEST | od -tx1 | xxd -r

Of course, echo TEST is just a placeholder here for eg. cat test.bmp or anything else.

> echo TEST
TEST

> echo TEST | od -tx1
0000000 54 45 53 54 0a
0000005 

> echo TEST | od -tx1 | xxd -r
TEST

That looks right, but it is different, as we can see here if we give it to od again:

> echo TEST | od -tx1 | xxd -r | od -tx1
0000000 54 45 53 54 0a 00 00 00
0000010 

Why does xxd -r add those 00 s?

Answer 1

You're getting those three nul bytes because of xxd -r trying and failing to parse input that's in a different format than what it expects. od -tx1 adds an extra line with an offset but no data bytes. Plus the offsets in xxd have a colon after them, and are printed with a different width, and printable bytes are displayed as well as the hex dump, and possibly in a different base... Something about that doesn't play well with xxd , and it adds the extra bytes as a result.

Examples:

$ echo TEST | xxd
00000000: 5445 5354 0a                             TEST.
$ echo TEST | xxd | xxd -r
TEST
$ echo TEST | xxd | xxd -r | xxd    
00000000: 5445 5354 0a                             TEST.
$ echo TEST | xxd | xxd -r | od -tx1
0000000 54 45 53 54 0a
0000005
$ echo TEST | od -tx1 | head -1 | xxd -r | od -tx1 
0000000 54 45 53 54 0a
0000005

See how they're not present when giving xxd -r its expected xxd style input? And how they're not there when you prune that extra line from od 's output? Don't mix and match incompatible data formats.

Answer 2

It seems to work if I remove the offsets at all:

> echo TEST | od -tx1 -An
 54 45 53 54 0a
> echo TEST | od -tx1 -An | xxd -r -p
TEST
> echo TEST | od -tx1 -An | xxd -r -p | od -tx1 -An
 54 45 53 54 0a

Bingo! Note the extra " " in front of the bytes. It seems to have no effect.