How do I read the Windows NTFS $Secure file (and/or the $SDS stream) programmatically in C#

Question

The methods in the .NET platform's DirectorySecurity namespace (eg GetAccessRules()) are far too slow for my purposes. Instead, I wish to directly query the NTFS $Secure metafile (or, alternatively, the $SDS stream) in order to retrieve a list of local accounts and their associated permissions for each file system object.

My plan is to first read the $MFT metafile (which I've already figured out how to do) - and then, for each entry therein, look up the appropriate security descriptor in the metafile (or stream).

The ideal code block would look something like this:

//I've already successfully written code for MFTReader:
var mftReader = new MFTReader(driveToAnalyze, RetrieveMode.All);
IEnumerable<INode> nodes = mftReader.GetNodes(driveToAnalyze.Name);

foreach (NodeWrapper node in nodes)
{
    //Now I wish to return security information for each file system object
    //WITHOUT needing to traverse the directory tree.
    //This is where I need help:
    var securityInfo = GetSecurityInfoFromMetafile(node.FullName, node.SecurityID);
    yield return Tuple.Create(node.FullName, securityInfo.PrincipalName, DecodeAccessMask(securityInfo.AccessMask));
}

And I would like my output to look like this:

c:\Folder1\File1.txt    jane_smith  Read, Write, Execute
c:\Folder1\File1.txt    bill_jones  Read, Execute
c:\Folder1\File2.txt    john_brown  Full Control
etc.

I am running .NET version 4.7.1 on the Windows 10.

Answer 1

There's no API to read directly from $Secure, just like there is no API to read directly from $MFT. (There's FSCTL_QUERY_FILE_LAYOUT but that just gives you an abstracted interpretation of the MFT contents.)

Since you said you can read $MFT, it sounds like you must be using a volume handle to read directly from the volume, just like chkdsk and similar tools. That allows you to read whatever you want provided you know how to interpret the on-disk structures. So your question reduces to how to correctly interpret the $Secure file.

I will not give you code snippets or exact data structures, but I will give you some very good hints. There are actually two approaches possible.

The first approach is you could scan forward in $SDS. All of the security descriptors are there, in SecurityId order. You'll find there's at various 16-byte aligned offsets, there will be a 20-byte header that includes the SecurityId among other information, and following that there's the security descriptor in serialized form. The SecurityId values will appear in ascending order in $SDS. Also every alternate 256K region in $SDS is a mirror of the previous 256K region. To cut the work in half only consider the regions 0..256K-1, 512K..768K-1, etc.

The second approach is to make use of the $SII index, also part of the $Secure file. The structure of this is a B-tree very similar to how directories are structured in NTFS. The index entries in $SII have SecurityId as the index for lookups, and also contain the byte offset you can go to in $SDS to find the corresponding header and security descriptor. This approach will be more performant than scanning $SDS, but requires you to know how to interpret a lot more structures.

Answer 2

Craig pretty much covered everything. I would like to clear some of them. Like Craig, no code here.

1. Navigate to the node number 9 which corresponds to $Secure.
2. Get all the streams and get all the fragments of the $SDS stream.
3. Read the content and extract each security descriptor.
4. Use IsValidSecurityDescriptor to make sure the SD is valid and stop when you reach an invalid SD.

Remember that the $Secure store the security descriptors in self-relative format .

Answer 3

Are you using FSCTL_QUERY_FILE_LAYOUT? The only real source of how to use this function I have found is here: https://wimlib.net/git/?p=wimlib;a=blob;f=src/win32_capture.c;h=d62f7d07ef20c08c9bec93f261131033e39b159b;hb=HEAD

It looks like he solves the problem with security descriptors like this: He gets basically all information about files from the MFT, but not security descriptors. For those he gets the field SecurityId from the MFT and looks in a hash table whether he already has a mapping from this ID to the ACL. If he has, he just returns it, otherwise he uses NtQuerySecurityObject and caches it in the hash table. This should drastically reduce the amount of calls. It assumes that there are few security descriptors and that the SecurityID field correctly represents the single instancing of the descriptors