更快地搜索大型csv文件中的字符串C＃

Question

I am having

• a DataTable (columns are AccId and TerrName) which contains more than 2000 rows.
• a large csv file (columns are AccId and External_ID) containing more than 6 millions records.

Now, I need to match AccId and have to find its corresponding External_ID from the csv file.

Currently I am achieving it using below code:

DataTable tblATL = Util.GetTable("ATL", false);
tblATL.Columns.Add("External_ID");

DataTable tbl = Util.CsvToTable("TT.csv", true);

foreach (DataRow columnRow in tblATL.Rows)
{
    var query = tbl.Rows.Cast<DataRow>().FirstOrDefault(x => x.Field<string>("AccId") == columnRow["AccId"].ToString());
    if (query != null)
    {
        columnRow["External_ID"] = query.Field<string>("External_ID");
    }
    else
    {
        columnRow["External_ID"] = "New";
    }
}

This code is working well but only problem is a performance issue, its taking very very long time to get the result.

Please help. How can I improve its performance, do you have any other approach?

Answer 1

I suggest organizing data into a dictionary , say, Dictionary<String, String[]> which has O(1) time complexity, eg

  Dictionary<String, String[]> Externals = File
    .ReadLines(@"C:\MyFile.csv")
    .Select(line => line.Split(',')) // the simplest, just to show the idea
    .ToDictionary(
      items => items[0], // let External_ID be the 1st column
      items => items // or whatever record representation
    );

  ....

  String externalId = ...

  String[] items = Externals[externalId];

EDIT : if same External_ID can appear more than once (see comments below) you have to deal with duplicates, eg

 var csv =  File
   .ReadLines(@"C:\MyFile.csv")
   .Select(line => line.Split(',')) // the simplest, just to show the idea

 Dictionary<String, String[]> Externals = new Dictionary<String, String[]>();

 foreach (var items in csv) {
   var key = items[0]; // let External_ID be the 1st column
   var value = items;  // or whatever record representation

   if (!Externals.ContainsKey(key)) 
     Externals.Add(key, value);
   // else {
   //   //TODO: implement, if you want to deal with duplicates in some other way 
   //}
 }