关于如何在C＃中从服务器访问大量文件的说明

Question

Basically I want to access 1000s of textfiles, input their data, store them in sqlite databases, parse them then show the output to users. So far I've developed a program that does this for only ONE textfile.

What I want to do: There is a Directory on our server which has about 15 folders. In each folder there are about 30-50 textfiles. I want to Loop through EACH FOLDER, and in each folder, loop through EACH file. A nice user helped me with doing this for 1000s of textfiles but I needed further clarification his method. This was his approach:

 private static void ReadAllFilesStartingFromDirectory(string topLevelDirectory)
{
    const string searchPattern = "*.txt";
    var subDirectories = Directory.EnumerateDirectories(topLevelDirectory);
    var filesInDirectory = Directory.EnumerateFiles(topLevelDirectory, searchPattern);

    foreach (var subDirectory in subDirectories)
    {
        ReadAllFilesStartingFromDirectory(subDirectory);//recursion
    }

    IterateFiles(filesInDirectory, topLevelDirectory);
}

private static void IterateFiles(IEnumerable<string> files, string directory)
{
    foreach (var file in files)
    {
        Console.WriteLine("{0}", Path.Combine(directory, file));//for verification
        try
        {
            string[] lines = File.ReadAllLines(file);
            foreach (var line in lines)
            {
                //Console.WriteLine(line);   
            }
        }
        catch (IOException ex)
        {
            //Handle File may be in use...                    
        }
    }
}

My problems/questions:

1) topLevelDirectory - what should I exactly put there? The 15 folders are located on a server with the format something like this \\servername\\randomfile\\random\\locationoftopleveldirectory. But how can I put the double slashes (at the begining of the path name) in this? Is this possible in c#? I thought we could only access local files (example :"c:\\" - paths with single, not double slashes)

2) I dont understand what the purpose of the first foreach loop is. "readAllFilesStartingFromDirectory(subDirectory)" , yes we are looping the folders, but we aren't even doing anything with that loop. It's just reading the folders.

Answer 1

I'm not going to know your top level directory, but essentially if your files are in C:\\tmp, then you would pass it @"C:\\tmp". Escape your string with the @ character to get double-slashes (or escape each slash individually).

string example0 = @"\\\\some\\network\\path";
string example1 = "\\\\\\\\some\\\\network\\\\path";

With ReadAllFilesStartingFromDirectory you're recursively calling IterateFiles, and it's doing whatever IterateFiles does in each directory. With the code you pasted above, that happens to be doing nothing, since Console.Writeline(line) is commented out.

Answer 2

Lets get to clarify the topLevelDirectory : This is a folder, which has items in it. It does not matter if these are files or other directories. These caontained other "subfolders" can contain folders themselves.

What toplevelDirectory means to you: take the folder which encapsulates all your files you need at the lowest level possible.

Your toplevelfolder is the directory which contains the 15 folders you want to crawl.

ReadAllFilesStartingFromDirectory(string topLevelDirectory) You need to realise what recursion means. Recursion describes a method which calls itself. Compare the name of the function (ReadAllFilesStartingFromDirectory), with the name of the function called in the foreach loop - they are the same.

In you case: The method gets all folders located in your topfolder. He then loops through all subfolders. Each subfolder then becomes the toplevel folder, which in turn can contain subfolders, who will become toplevelfolders in the next method call. This is a nice way to loop through the whole file structure. If there are no more subfolders, there won't be any recursion and the method ends.

Your path problem: You need to mask the backslashes. You mask them by adding a backslash in front of them.

\\path\\randfolder\\file.txt will become \\\\path\\\\randfolder\\\\file.txt

Or you set an @ before the string. var path = @"\\path\\randfolder\\file.txt" , which also does the trick for you. Both ways work

Answer 3

1) Yes it is possible in C#. If your program has access permission to the network location you can use: "\\\\\\\\servername\\\\randomfile\\\\random\\\\locationoftopleveldirectory" - double slash in string interpreated as one slash. or you can use @ before the string, it means 'ignore escape character' which is slash, then your string will look like this: @"\\\\servername\\randomfile\\random\\locationoftopleveldirectory"

2) ReadAllFilesStartingFromDirectory is recursive function. Directories structure is hierarchical, therefore it is easy to traverse them recursively. This function looks for files in the root directory and in its sub-directories and in all their sub directories... Try to put comment on this loop and you will see that only the files of the root directory parsed by the IterateFiles function