Comparing two large generic lists

Question

I cannot find a specific example of this, so am posting the question. Any help appreciated.

I have two large generic lists, both with over 300K items.

I am looping through the first list to pull back information and generate a new item for a new list on the fly, but I need to search within the second list and return a value, based on THREE matching criteria, if found to add to the list, however as you can imagine, doing this 300k * 300k times is taking time.

Is there any way I can do this more efficiently?

My code:

var reportList = new List<StocksHeldInCustody>();
foreach (var correctDepotHolding in correctDepotHoldings)
  {
    var reportLine = new StocksHeldInCustody();
    reportLine.ClientNo = correctDepotHolding.ClientNo;
    reportLine.Value = correctDepotHolding.ValueOfStock;
    reportLine.Depot = correctDepotHolding.Depot;
    reportLine.SEDOL = correctDepotHolding.StockCode;
    reportLine.Units = correctDepotHolding.QuantityHeld;
    reportLine.Custodian = "Unknown";
    reportLine.StockName = correctDepotHolding.StockR1.Trim() + " " + correctDepotHolding.StockR2.Trim();

    //Get custodian info

    foreach (var ccHolding in ccHoldList)
    {
      if (correctDepotHolding.ClientNo != ccHolding.ClientNo) continue;
      if (correctDepotHolding.Depot != ccHolding.Depot) continue;
      if (correctDepotHolding.StockCode != ccHolding.StockCode) continue;
      if (correctDepotHolding.QuantityHeld != ccHolding.QuantityHeld) continue;
      reportLine.Custodian = ccHolding.Custodian;
      break;
    }
    reportList.Add(reportLine);
  }

Answer 1

As Pranay says, a join is probably what you want:

var query = from correct in correctDepotHoldings
            join ccHolding in ccHoldList
              on new { correct.ClientNo, correct.Depot,
                       correct.StockCode, correct.QuantityHeld }
              equals new { ccHolding.ClientNo, ccHolding.Depot,
                           ccHolding.StockCode, ccHolding.QuantityHeld }
            // TODO: Fill in the properties here based on correct and ccHolding
            select new StocksHeldInCustody { ... };
var reportList = query.ToList();

Answer 2

You could move the data from the lookup list into a dictionary, with the key being a unique hash of the 3 items you are searching on. Then you will have very quick lookups and save millions of iterations.

Answer 3

Check my full post : Linq Join on Mutiple columns using Anonymous type

Make use of Linq inner join that will do work for you.

var list =  ( from x in entity
           join y in entity2 
          on new { x.field1, x.field2 } 
        equals new { y.field1, y.field2 }
        select new entity { fields to select}).ToList();

Join of linq on multiple field

EmployeeDataContext edb= new EmployeeDataContext();
var cust = from c in edb.Customers
           join d in edb.Distributors on
             new { CityID = c.CityId, StateID = c.StateId, CountryID = c.CountryId,
                   Id = c.DistributorId }   
           equals
             new { CityID = d.CityId, StateID = d.StateId, CountryID = d.CountryId,
                   Id = d.DistributorId }   
           select c;

Answer 4

Use LINQ to join the lists and return it how you like.

eg

var list1 = GetMassiveList();
var list2 = GetMassiveList();

var list3 = from a in list1
            join b in list2
                 on new { a.Prop1, a.Prop2 } equals
                    new { b.Prop1, b.Prop2 } 
            select new { a.Prop1, b.Prop2 };

To do your outter join, you can use DefaultIfEmpty() This example is setting your RIGHT part of the join to a default object (often null) for the cases where a join wasn't made.

eg

from a in list1
join b in list2
    on new { a.Prop1, a.Prop2 } equals
       new { b.Prop1, b.Prop2 } 
into outer
from b in outer.DefaultIfEmpty()
select new 
    Prop1 = a.Prop1,
    Prop2 = b != null ? b.Prop2 : "Value for Prop2 if the b join is null"
}