bash - hash binary contents of variable without creating a file

Question

I am trying to obtain the hash of the contents stored in a variable created from a curl statement without outputting the curl into a file.

Basically I am trying to avoid:

curl website.com/file.exe >> tempfile.bin
md5sum tempfile.bin

The above provides the correct md5 hash. My approach (below) seemed to have work when testing plain text files, however when I attempted to download a binary and save it to a variable the hash was different than when I saved it to a file.

My attempt:

binary=$(curl website.com/file.exe)
echo $binary | md5sum

I think I may be missing a flag, or perhaps echo may not be the best way to do it. The important part of this challenge is not writing a file to disk, yet achieving the same md5 hash as if it were written to disk.

Answer 1

To also skip the step of using a temp variable, you can use process substitution:

md5sum <(curl website.com/file.exe)

or pipe to md5sum directly:

curl website.com/file.exe | md5sum

Answer 2

The bash shell doesn't handle raw binary data well, as you've experienced. To accomplish your goal, you will need to encode the file contents into a text format when you read them into the bash variable, and decode them when you write them out.

For instance, if you have the base64 tool, you can use it to re-implement your example like this:

encoded=$(curl website.com/file.exe | base64)
echo "$encoded" | base64 --decode | md5sum

If you later want to save the data to a file named $output, you could do it like this:

echo "$encoded" | base64 --decode -o "$output"