loading a text file for quick access C# windows form app

Question

I couldn't find an answer to this on the other stackoverflow is I'll ask it here. I have a text file with roughly 100000 rows. I have been doing multiple queries on it such as this

string header = isFirstRowHeader ? "Yes" : "No";
string pathOnly = Path.GetDirectoryName(path);
string fileName = Path.GetFileName(path);
string sql = @"SELECT Count(*) as NumberofRecords FROM [" + fileName + "]";

using (OleDbConnection connection = new OleDbConnection(
       @"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + pathOnly +
       ";Extended Properties=\"Text;HDR=" + header + "\""))
using (OleDbCommand command = new OleDbCommand(sql, connection))
using (OleDbDataAdapter adapter = new OleDbDataAdapter(command))
{
    DataTable dt = new DataTable();
    dt.Locale = CultureInfo.CurrentCulture;
    adapter.Fill(dt);
    return dt;
}

And it is doing multiple versions of this. Is there any way to load the textfile so that I can run things like this faster? Is there a better way? Currently it is taking too long.

Answer 1

What are you trying to do?

From your sample it looks like the only thing you're trying to do is to get the number of records in the file.

You might be safe to instead just count the number of lines (-1 line for the header) iff* you don't have content that spans multiple lines.

* if, and only if

EDIT:

So counting the number of lines is not an option since you're doing more complicated stuff.

I just generated a sample file with 100k records (7.7 MB in size) which got processed in 0.43 seconds. Doing a count(*) .. group by Name took 0.58 seconds.

What are your numbers and why do you think it's taking too long? Where does the file lie? Is it perhaps a network/slow drive issue?

Answer 2

Load the file into memory using a stream, See here . Once it's in memory, run your queries etc.

Answer 3

You can use the the following example:

string filename = @"C:\BigTextFile.txt";  
StreamReader sr = System.IO.File.OpenText(filename);

// Process line by line.  
string line = "";  
do  
{  
line = sr.ReadLine();  
}  
while(sr.Peek() != -1);  

// Load all at once and process.  
string alltext = sr.ReadToEnd();  

sr.Close();