仅使用sed打印二进制匹配项？

Question

Let's first have a binary test file:

echo -e '\x00\x01\x00\x0a\x00\x0f\x32\x7a\xb0\x00\x00\x01' > test.bin

hexdump -C test.bin 
# 00000000  00 01 00 0a 00 0f 32 7a  b0 00 00 01 0a           |......2z.....|
# 0000000d

Now let's see if I can match the bytes' sequence 0x0f 0x32 0x7a with sed :

sed -n 's/\(\x0f\x32\x7a\)/\1/p' test.bin | hexdump -C
# 00000000  00 0f 32 7a b0 00 00 01  0a                       |..2z.....|
# 00000009

That works as expected - the printed match is from the last linefeed 0x0a to the next one end. Now, I want to print the match only - first I try to filter out with .* regex at start:

sed -n 's/.*\(\x0f\x32\x7a\)/\1/p' test.bin | hexdump -C
# 00000000  0f 32 7a b0 00 00 01 0a                           |.2z.....|
# 00000008

That works - now let's do the same, but also for the trailing part:

sed -n 's/.*\(\x0f\x32\x7a\).*/\1/p' test.bin | hexdump -C
# 00000000  0f 32 7a b0 00 00 01 0a                           |.2z.....|
# 00000008

Well, that does not work - only the heading part is removed - but the trailing part keeps going, even if I also terminated my sed match pattern with .* ??!

What is going on here - and how can I get sed to print out only the bytes 0x0f 0x32 0x7a on output (taking into account that hexdump sed will add the final linefeed 0x0a, when it prints a match)?

Answer 1

Interesting. Here's a simpler repro case:

echo -en '\xff\x80' | sed -n 's/\xff.*/!/p' | hexdump -C

The above prints 21 80 which is !\\x80 . The \\x80 can be a larger ASCII code too, but it cannot be smaller: \\x7F has sed doing the expected thing, printing only the ! .

Also check out what this does:

echo -en '\xff\x80' | sed -n 's/\xff./!/p' | hexdump -C

It prints nothing at all.

So the question becomes, what's special about \\x80 and higher? Well, UTF-8 of course! In UTF-8, having the first bit of a code point set indicates more bytes are coming. And sed never finds them, so it never interprets the character at all.

If you want to "fix" it, tell sed to use the "good old" C locale:

LC_ALL=C sed ...

Then you get your expected output:

echo -e '\x00\x01\x00\x0a\x00\x0f\x32\x7a\xb0\x00\x00\x01' |
  LC_ALL=C sed -n 's/.*\(\x0f\x32\x7a\).*/\1/p' |
  hexdump -C

00000000  0f 32 7a 0a                                       |.2z.|