Fastest way to delete files that are not in a data table?

Question

I need to write a code in C# that will select a list of file names from a data table and delete every file in a folder that is not in this list.

One possibility would be to have both ordered by name, and then loop through my table results, and for each result, loop through my files and delete them until I find a file that matches the current result or is alphabetically bigger, and then move to the next result without resetting the current file index.

I haven't tried to actually implement this, but seems to me that this would be an O(n) since each list would be looped through just once (ignoring the sorting both lists part). The only thing I'm not sure about is whether I can be 100% sure both the file system and the database engine will sort exactly the same way (will they both consider "_" smaller than "-" and stuff like that). If not, the algorithm above just wouldn't work at all. (By the way this is a Jet Engine database.)

But since this is probably not such an uncommon problem you guys might already know a better solution. I tried search the web but couldn't find anything. Perhaps a more effective solution would be to put each list into a HashSet and find their difference.

Answer 1

1. Get the folder content into folderFiles ( IEnumerable<string> )
2. Get the file you want to keep in filesToKeep ( IEnumerable<string> )
3. Get a list of "not in list" files.
4. Delete these files.

Code Sample :

IEnumerable<FileInfo> folderFiles = new List<FileInfo>(); // Fill me.
IEnumerable<string> filesToKeep = new List<string>();     // Fill me.
foreach (string fileToDelete in folderFiles.Select(fi => fi.FullName).Except(filesToKeep))
{
    File.Delete(fileToDelete);
}

Answer 2

Here is my suggestion for you. Assuming filesInDatabase contains a list of files which are in the database and pathOfDirectory contains the path of the directory where the files to compare are contained.

foreach (var fileToDelete in Directory.EnumerateFiles(pathOfDirectory).Where(item => !filesInDatabase.Contains(item))
{
    File.Delete(fileToDelete);
}

EDIT:

This requires using System.Linq; , because it uses LINQ.

Answer 3

I think hashing is the way to go, but you don't really need two HashSets. Only one HashSet is needed to store the standardized file names from the datatable; the other container can be any collection data type.

Answer 4

First off, .Net allows you to define cultures that can be used in sorting, but I'm not all that familiar with the mechanism, so I'll let Google to give his pointers on the subject.

Second, to avoid all the culture mass, you can use a different algorithm with an idea similar to radix-sort (only without the sort) - time complexity is O(n * length_longest_file_name). File name lengths are limited (as far as I know, almost no file system will allow a file name longer then 256), so I'm assuming that n is dramatically larger then file name lengths, and if n is smaller then the max file name length, just use an O(n^2) method and avoid the work (iterating lists this small is near instant times anyways). Note: This method does not require sorting.

The idea is to create an array of symbols that can be used as file name chars (about 60-70 chars, if this is a case sensitive search), and another flag array with a flag for each char in the first array. Now, you create a loop for each char in the file names of the list from the DB (from 1 -> length_longest_file_name). In each iteration (i) you go over the i-th char of each file name in the DB list. Every char you see, you set it's relevant flag to true. When all flags are set, you go over the second list and delete every file for which the i-th char of it's name is not flagged.

Implementation might be complex, and the overhead of the two arrays might make it slower when n is small, but you can optimize this to make it better (for instance, no iterating over files that have names shorter then the current i by removing them from both lists).

Hope this helps

Answer 5

I have another idea that might be faster.

var filesToDelete = new List<string>(Directory.GetFiles(directoryPath));
foreach (var databaseFile in databaseFileList)
{
    filesToDelete.Remove(databaseFile);
}
foreach (var fileToDelete in filesToDelete)
{
    File.Delete(fileToDelete);
}

Explanation: First get all files containing in the directory. Then delete every file from that list, which is in the database. At last delete all remaining files from the list filesToDelete.