macos中目录的阴影

Question

I'm writing a shell script that uses the shasum to check if the contents of a directory have changed.

On Linux and FreeBSD, the shasum have the same behavior when I do shasum <directory> however, on MacOS the shasum give me hashes for files only .

FreeBSD

$ shasum CONTENTS/
7f986e5e5289c59db1bba48df92ffe4707830aaa  CONTENTS/

Linux

$ shasum CONTENTS/
7f986e5e5289c59db1bba48df92ffe4707830aaa  CONTENTS/

MacOS

$ shasum CONTENTS/
shasum: CONTENTS/: 

How could I calculate the hash of a directory in MacOS?

TRY 1: Using TAR with pipes

Tried to use but seems that this tar option doesn't work on MacOS.

tar cO CONTENTS/ | shasum
tar: Option -O is not permitted in mode -c
da39a3ee5e6b4b0d3255bfef95601890afd80709  -

TRY 2: Using FIND/EXEC

It was consistent between MacOS and FreeBSD, but Linux returned a weird hash

find CONTENTS -type f -exec shasum {} \; | sort -k 2 | shasum

Linux

c2ddb9bc5f543e956f5cdcc76750cb78cc5f26f3

FreeBSD

3ac2a9d4e2fc5d2d2ec3c7f612e680990cc35824

MacOS

3ac2a9d4e2fc5d2d2ec3c7f612e680990cc35824

OTHER FINDINGS ON TAR tar would be excellent as it "archives" a folder and then I could shasum it, however the order of how tar "walk" the folder structure is not consistent across operating systems . As some helpers mentioned in the comments that I should use the same version of tar in all systems.

Just an example, on system 1 I have this order:

drwxr-xr-x  0 root   wheel       0 27 Jul 07:23 usr/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f1/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f1/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f1/f0/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f1/f0/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f2/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f2/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/f0/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/f0/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f3/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f3/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/f1/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/f1/aaa

and on system 2 I have the following order:

drwxr-xr-x  0 root   wheel       0 27 Jul 07:23 usr/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f1/
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f2/
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f3/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f3/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/f1/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f3/f2/f1/aaa
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f2/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/f0/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f2/f1/f0/aaa
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f1/aaa
drwxr-xr-x  0 root   wheel       0 27 Jul 07:25 usr/f1/f0/
-rw-r--r--  0 root   wheel       0 27 Jul 07:25 usr/f1/f0/aaa

From a tar standpoint it if all good, but due to the order, the shasum produces a different hash.

CONCLUSION

shasum is consistent among Linux and BSDs to check an individual file hash, but, when it comes to directories the consistency happens only on MacOS and FreeBSD, perhaps due to how files are sorted.

If sorting is enforced using the find command, consistency is only obtained in FreeBSD and MacOS, however this method is time prohibitive as it takes a significant amount of time to calculate the hashes for every single file and then the whole structure hash.

Using tar to create a temporary file and then doing a shasum also found to be inconsistent between Linux and BSDs, perhaps because of difference in the archiving method.

I think the only way forward is to redesign my solution .

Answer 1

I had this problem some time ago and I ended up tarring the directory and generating a hash from the tar.

$ mkdir -p test
$ echo 1 > test/tmp
$ tar cO test/ | md5sum
7b18a99a8ccfef1ebbfd1e7a8b2852ee
$ echo 2 > test/tmp
$ tar cO test/ | md5sum
644042dd530157e604641ea89b4e9152

Note that if you write the same content to a file, the modified stat is updated and a new hash will be generated for that directory.

$ echo 2 > test/tmp
$ tar cO test/ | md5sum
da25819594f123563a837d5786e51950  -
$ echo 2 > test/tmp 
$ tar cO test/ | md5sum
9407b64d43b809a5828a9fc2297b4e9c  -

ps you should change md5sum for shasum :-)

EDIT:
Discussed it shortly with a friend who uses MacOSX and he came up with:

$ mkdir tmp; echo 1 > test/a;
$ tar -cf - tmp/ | md5sum
26c43adc9eca9f63279d08a0d145dd7d  - 
$ echo 2 > test/a
$ tar -cf - tmp/ | md5sum
dc3a68cac0b0224be9b202d86e69c5bd  -

Note that MACosx uses md5 instead of md5sum .