Filtering rows from DataTable

Question

I have a DataTable with 1000000 records. I am filtering records from DataTable . I have used various methods for filtering:

• DataTable.Select()
• Linq to DataTable
• DataView.RowFilter

After filtering records, binding the resulting records (approx. 1000 records) to a ListBox.

The whole process is taking much time. I analyzed the code and found that filtering is taking much time. The whole process should be done in milliseconds. For each filtering I can't hit on server to fetch records.

Please suggest.

//using dataTable
DataRow[] dr = dtData.Select("i_pernr=" + perno + "");
dr.CopyToDataTable(dtTemp,LoadOption.OverwriteChanges);

//using DataView
dtData.DefaultView.RowFilter = "i_pernr=" + perno ;
dtTemp = dtData.DefaultView.ToTable();

//Using Linq
IEnumerable<DataRow> query = from m in dtData.AsEnumerable()
                             where m["i_pernr"] == perno
                             select m;
query.CopyToDataTable(dtTemp,LoadOption.OverwriteChanges);

Answer 1

If you really feel that filtering at the source isn't possible (SQL Server is much better at selecting records) you can only speed up the selection operation by setting the PrimaryKey for the DataTable .

dtData.PrimaryKey = new DataColumn[] { dtData.Columns["i_pernr"]};

Notice that you can have multiple columns as primarykey columns. You'll have to add as much as needed to make guarantee the datarows you add/have are unique.

I've used the following test rig:

var dtTemp = new DataTable();
dtTemp.Columns.Add("i_pernr", typeof(int));
dtTemp.Columns.Add("name", typeof(string));

var dtData = new DataTable();
dtData.Columns.Add("i_pernr", typeof(int));
dtData.Columns.Add("name", typeof(string));
dtData.PrimaryKey = new DataColumn[] { 
    dtData.Columns["i_pernr"], 
    dtData.Columns["name"]
    };

var rnd = new Random();
for(int r = 1; r<1000000; r++)
{
    var row =dtData.NewRow();
    row[0] = rnd.Next(1000);
    row[1]= String.Format("the lazy fox jumps again {0}",rnd.Next(10000000)) ;
    try
    {
       dtData.Rows.Add(row);
    }
    catch
    {
       // Hey, for testing this is fine ...
    }
}

This loads a datatable with enough rows to test the effect of having or not having a PrimaryKey it the stopwatch:

dtData.Rows.Count.Dump("rows");

var perno = rnd.Next(1000).ToString();

var sw = new Stopwatch();
sw.Start();
DataRow[] dr = dtData.Select("i_pernr=" + perno + "");
dr.CopyToDataTable(dtTemp,LoadOption.OverwriteChanges);
sw.Stop();
sw.ElapsedMilliseconds.Dump("elapsed");
sw.ElapsedTicks.Dump("ticks");

This output in LinqPad when using the PrimaryKey:

elapsed:    0 
ticks:  5,371

Without the primarykey the results are:

elapsed:     2,444 
ticks:   7,541,415