How do remote file “stubs” work on Macs with APFS?

Question

Both iCloud Drive and OneDrive for Mac support remote files (or "Files On-Demand" as OneDrive calls it ).

• These files appear in Finder and seem to behave as real files (with odd Quick Look behavior).
• The Inspector shows their true size plus (4 KB on disk) or similar.

Also:

• Terminal does not seem to understand iCloud remote size:

: ~/Library/Mobile Documents/com~apple~Preview/Documents
$ ls -lh
total 24
-rw-r--r--@ 1 REDACTED  staff   157B Apr 28  2014 .2AM.png.icloud
-rw-r--r--@ 1 REDACTED  staff   6.0K Sep 30  2018 .DS_Store

• But it understands OneDrive's (the file shows 0 bytes on disk in Finder):

: ~/OneDrive/Pictures/Drawings
$ ls -lh
total 24
-rwxrwxrwx@  1 REDACTED  staff    11M Apr  5  2019 MMM.kra*

Disappointingly, I cannot find any information about these files online. The official Apple File System Reference doesn't seem to show any clear endpoints for this.

• How do they work?
• Is this a public macOS API or feature of APFS? Is this dependent on APFS?

Answer 1

It looks like they are using sparse files (a feature of APFS) to show the size without using up space.

In order to populate the file on-demand, they are using extended file attributes.

Try this command on your files to see more info: ls -l@

I haven't seen any documentation on how to use this anywhere, but apparently there is an attribute you can set to get a callback when another application opens the file. This works in conjunction with the com.apple.fileutil kext, which OneDrive uses. Microsoft must have special access.

Answer 2

It's filecoordinationd which handles all this magic. The files are marked as dateless placeholders, and filecoordinationd is able to "materialize" them on access by retrieving them from iCloud (as well as handle the reverse direction when they are saved or "evicted".

There's an Apple utility in /usr/bin/fileproviderctl which can be used to explore this further. (eg fileproviderctl materialize..)

Sparse files are something entirely different: They're for zero-filled files which can then take up far less space as their zero-ed blocks take no space.

Reference: NewOSXBook.com, Volume II.