在多个文件中搜索文本的最快方法？

Question

I need to find some text in around 120 text files, I want to know which will be the best and fastest way to search text. Am I supposed to read each file in a RichTextBox then use its methods to search text or should I be reading those files into a string variable and then searching using regular expressions?

I think the major factor behind performance is to find a way so that there is no need to loop through the lines which have already been tested for match. Is there any way to find all the matches in a file in one go? Does anyone knows a way to find matches in text files as Visual Studio does? It searched 200 text files for match in around 800-1000 Milliseconds. I think it makes use of multiple threads to accomplish this.

Answer 1

From your description (120 files, 70K-80K words, 1-2 MB per file), it would seem the best approach is to read the files once and build an index that can be searched. I have included an example below to illustrate how such a thing can be done, but that may be of limited use to you if you need more complex search term matching than finding an exact term or a prefixed term.

If you need more complex text search matching (while getting good performance), I would advise you to look into the excellent Lucene library, that was built specifically for this purpose.

public struct WordLocation
{
    public WordLocation(string fileName, int lineNumber, int wordIndex)
    {
        FileName = fileName;
        LineNumber = lineNumber;
        WordIndex = wordIndex;
    }
    public readonly string FileName; // file containing the word.
    public readonly int LineNumber;  // line within the file.
    public readonly int WordIndex;   // index within the line.
}

public struct WordOccurrences
{
    private WordOccurrences(int nOccurrences, WordLocation[] locations)
    {
        NumberOfOccurrences = nOccurrences;
        Locations = locations;
    }

    public static readonly WordOccurrences None = new WordOccurrences(0, new WordLocation[0]);

    public static WordOccurrences FirstOccurrence(string fileName, int lineNumber, int wordIndex)
    {
        return new WordOccurrences(1, new [] { new WordLocation(fileName, lineNumber, wordIndex) });
    }

    public WordOccurances AddOccurrence(string fileName, int lineNumber, int wordIndex)
    {
        return new WordOccurrences(
            NumberOfOccurrences + 1, 
            Locations
                .Concat(
                    new [] { new WordLocation(fileName, lineNumber, wordIndex) })
                .ToArray());
    }

    public readonly int NumberOfOccurrences;
    public readonly WordLocation[] Locations;
}

public interface IWordIndexBuilder
{
    void AddWordOccurrence(string word, string fileName, int lineNumber, int wordIndex);
    IWordIndex Build();
}

public interface IWordIndex
{
    WordOccurrences Find(string word);
}

public static class BuilderExtensions
{
    public static IWordIndex BuildIndexFromFiles(this IWordIndexBuilder builder, IEnumerable<FileInfo> wordFiles)
    {
        var wordSeparators = new char[] {',', ' ', '\t', ';' /* etc */ };
        foreach (var file in wordFiles)
        {
            var lineNumber = 1;
            using (var reader = file.OpenText())
            {
                while (!reader.EndOfStream)
                {
                    var words = reader
                         .ReadLine() 
                         .Split(wordSeparators, StringSplitOptions.RemoveEmptyEntries)
                         .Select(f => f.Trim());

                    var wordIndex = 1;
                    foreach (var word in words)
                        builder.AddWordOccurrence(word, file.FullName, lineNumber, wordIndex++);

                    lineNumber++;
                }
            }
        }
        return builder.Build();
    }
}

Then the simplest index implementation (that can only do an exact match lookup) uses a dictionary internally:

public class DictionaryIndexBuilder : IIndexBuilder
{
    private Dictionary<string, WordOccurrences> _dict;

    private class DictionaryIndex : IWordIndex 
    {
        private readonly Dictionary<string, WordOccurrences> _dict;

        public DictionaryIndex(Dictionary<string, WordOccurrences> dict)
        {
            _dict = dict;
        }
        public WordOccurrences Find(string word)
        {
           WordOccurrences found;
           if (_dict.TryGetValue(word, out found);
               return found;
           return WordOccurrences.None;
        }
    }

    public DictionaryIndexBuilder(IEqualityComparer<string> comparer)
    {
        _dict = new Dictionary<string, WordOccurrences>(comparer);
    }
    public void AddWordOccurrence(string word, string fileName, int lineNumber, int wordIndex)
    {
        WordOccurrences current;
        if (!_dict.TryGetValue(word, out current))
            _dict[word] = WordOccurrences.FirstOccurrence(fileName, lineNumber, wordIndex);
        else
            _dict[word] = current.AddOccurrence(fileName, lineNumber, wordIndex);
    }
    public IWordIndex Build()
    {
        var dict = _dict;
        _dict = null;
        return new DictionaryIndex(dict);
    }
}

Usage:

var builder = new DictionaryIndexBuilder(EqualityComparer<string>.Default);
var index = builder.BuildIndexFromFiles(myListOfFiles);
var matchSocks = index.Find("Socks");

If you also want to do prefix lookups, implement an index builder/index class that uses a sorted dictionary (and change the IWordIndex.Find method to return multiple matches, or add a new method to the interface for finding partial/pattern matches).

If you want to do more complex lookups, go for something like Lucence.

Answer 2

Here what I will if I where you:

1- I'll load all files path to a list of string.

2- I'll create a new list to store files path that match my search term.

3- I'll loop foreach in Files list and search for my term then I'll add the matched file to the new list.

string searchTerm = "Some terms";
    string[] MyFilesList = Directory.GetFiles(@"c:\txtDirPath\", "*.txt");
    List<string> FoundedSearch=new List<string>();
    foreach (string filename in MyFilesList)
    {
        string textFile = File.ReadAllText(filename);
        if (textFile.Contains(searchTerm))
        {
            FoundedSearch.Add(filename);
        }
    }

then you can deal with the List:FoundedSearch what ever you want.

by the way:

I don't know the best answer but the performance will be very well until 800 text file with 1000 word per file you can find the performance pretty well with this chart

Answer 3

I assume you need to search every file for the same string. You could use a compiled regex for each search.

string searchTerm = "searchWord";
Regex rx = new Regex(String.Format("\b{0}\b", searchTerm), RegexOptions.Compiled);
List<string> filePaths = new List<string>();

foreach (string filePath in filePaths)
{
   string allText = File.ReadAllText(filePath);
   var matches = rx.Matches(allText);             
   //rest of code
}

You'd have to benchmark performance but I imagine the major bottleneck will be the opening and reading of the file from the disk. You could look into Memory-Mapped Files if that turns out to be the case. Alternatively, depending on what you're ultimately trying to do, a dedicated text searcher such as Lucene.Net (as I4V mentioned in the comments) might be more suitable.