File.ReadAllText OutOfMemoryException

Question

I have a large SQL script (around 70MB), and I need to read it all into memory, and then split on the GO statements.

I'm doing the following, but getting OutOfMemoryException :

var script = File.ReadAllText(scriptFile);
var scriptName = Path.GetFileName(scriptFile);
var commands = Regex.Split(script, "^GO\r\n", RegexOptions.Multiline | RegexOptions.IgnoreCase);

Then, for each script fragment in commands , I call ExecuteNonQuery . This all works fine until I try to load this large script file.

Any suggestions for parsing that text file more efficiently?

Answer 1

Try something like this

public static IEnumerable<string> SplitScriptOnGo(string scriptPath)
{
    var buffer = new StringBuilder();
    foreach (var line in File.ReadLines(scriptPath))
    {
        if (line == "GO")
        {
            yield return buffer.ToString();
            buffer.Clear();
        }
        else
        {
            buffer.AppendLine(line);
        }
    }
}

That will read one line at a time, buffer them into a StringBuilder and then yield the buffer when it sees a "Go" line. As long as you don't ToList this you can iterate over the results and execute each SQL statement without reading the entire script into memory.

Answer 2

Does it fail on ReadAllText() or on Split()?

I don't know your system specs, but I doubt you have insufficient memory, my guess is it fails on Split() and something in your split is causing an infinite loop.

Perhaps instead of reading everything at once you could simply read until the first "go", execute, then continue reading to next go, etc? Eventually it'll either process the whole file, or it'll fail (at which point you'll know which "GO" statement is causing the issues by looking at the string you just read in the debugger).