从文件中删除包含多个字符串的行的最有效方法？

Question

I want to find the most efficient way of removing string 1 and string 2 when reading a file (host file) and remove the entire lines that contains string 1 or string 2.

Currently I have, and is obviously sluggish. What better methods are there?

using(StreamReader sr = File.OpenText(path)){
    while ((stringToRemove = sr.ReadLine()) != null)
    {
        if (!stringToRemove.Contains("string1"))
        {
            if (!stringToRemove.Contains("string2"))
            {
                emptyreplace += stringToRemove + Environment.NewLine;
            }
        }
    }
    sr.Close();
    File.WriteAllText(path, emptyreplace);
    hostFileConfigured = false;
    UInt32 result = DnsFlushResolverCache();
    MessageBox.Show(removeSuccess, windowOffline);
}

Answer 1

The primary problem that you have is that you are constantly using large regular strings and appending data onto the end. This is re-creating the strings each time and consumes a lot of time and particularly memory. By using string.Join it will avoid the (very large number of) intermediate string values being created.

You can also shorten the code to get the lines of text by using File.ReadLines instead of using the stream directly. It's not really any better or worse, just prettier.

var lines = File.ReadLines(path)
    .Where(line => !line.Contains("string1") && !line.Contains("string2"));

File.WriteAllText(path, string.Join(Environment.NewLine, lines));

Another option would be to stream the writing of the output as well. Since there is no good library method for writing out a IEnumerable<string> without eagerly evaluating the input, we'll have to write our own (which is simple enough):

public static void WriteLines(string path, IEnumerable<string> lines)
{
    using (var stream = File.CreateText(path))
    {
        foreach (var line in lines)
            stream.WriteLine(line);
    }
}

Also note that if we're streaming our output then we'll need a temporary file, since we don't want to be reading and writing to the same file at the same time.

//same code as before
var lines = File.ReadLines(path)
    .Where(line => !line.Contains("string1") && !line.Contains("string2"));

//get a temp file path that won't conflict with any other files
string tempPath = Path.GetTempFileName();
//use the method from above to write the lines to the temp file
WriteLines(tempPath, lines);
//rename the temp file to the real file we want to replace, 
//both deleting the temp file and the old file at the same time
File.Move(tempPath, path);

The primary advantage of this option, as opposed to the first, is that it will consume far less memory. In fact, it only ever needs to hold line of the file in memory at a time, rather than the whole file. It does take up a bit of extra space on disk (temporarily) though.

Answer 2

The first thing that shines to me, is wrong (not efficient) use of string type variable inside a while loop ( emptyreplace ), use StrinBuilder type and it will be much memory efficient.

For example:

 StringBuilder emptyreplace = new StringBuilder(); 

using(StreamReader sr = File.OpenText(path)){
    while ((stringToRemove = sr.ReadLine()) != null)
    {
        if (!stringToRemove.Contains("string1"))
        {
            if (!stringToRemove.Contains("string2"))
            {
                //USE StringBuilder.Append, and NOT string concatenation
                emptyreplace.AppendLine(stringToRemove + Environment.NewLine);
            }
        }
    }
   ...
}

The rest seems good enough.

Answer 3

Two suggestions:

1. Create an array of strings to detect (I'll call them stopWords ) and use Linq's Any extension method.

2. Rather than building the file up and writing it all at once, write each line to an output file one at a time while your reading the source file, and replace the source file once your done.

The resulting code:

string[] stopWords = new string[]
{
    "string1",
    "string2"
}

using(StreamReader sr = File.OpenText(srcPath))
using(StreamWriter sw = new StreamWriter(outPath))
{
    while ((stringToRemove = sr.ReadLine()) != null)
    {
        if (!stopWords.Any(s => stringToRemove.Contains(s))
        {
            sw.WriteLine(stringToRemove);
        }
    }
}

File.Move(outPath, srcPath);

Answer 4

There are a number of ways to improve this:

• Compile the array of words you're searching for into a regex (eg, word1|word2 ; beware of special characters) so that you'll only need to loop over the string once. (this would also allow you to use \\b to only match words)

• Write each line through a StreamWriter to a new file so that you don't need to store the whole thing in memory while building it. (after you finish, delete the original file & rename the new one)

Answer 5

Is your host file really that big that you need to bother with reading it line by line? Why not simply do this?

var lines = File.ReadAllLines(path);
var lines = lines.Where(x => !badWords.Any(y => x.Contains(y))).ToArray();
File.WriteAllLines(path, lines);

Answer 6

Update : I just realized that you are actually talking about the "hosts file". Assuming you mean %windir%\\system32\\drivers\\etc\\hosts , it is very unlikely that this file has a truly significant size (like more than a couple of KBs). So personally, I would go with the most readable approach. Like, for example, the one by @servy .

In the end you will have to read every line and write every line, that does not match your criteria. So, you will always have the basic IO overhead that you cannot avoid. Depending on the actual (average) size of your files that might overshadow every other optimization technique you use in your code to actually filter the lines.

Having that said, you can however be a little less wasteful on the memory side of things, by not collecting all output lines in a buffer, but directly writing them to the output file as you have read them (again, this might be pointless if you files are not very big).

using (var reader = new StreamReader(inputfile))
{
  using (var writer = new StreamWriter(outputfile))
  {
    string line;
    while ((line = reader.ReadLine()) != null)
    {
       if (line.IndexOf("string1") == -1 && line.IndexOf("string2") == -1)
       {
          writer.WriteLine(line);
       }
    }
  }
}

File.Move(outputFile, inputFile);