C# Zip Function to iterate through two lists of objects

Question

I have a program that creates a list of objects from a file, and also creates a list of the same type of object, but with fewer/and some different properties, from the database, like: List from FILE: Address ID, Address, City, State, Zip, other important properties list from DB: Address ID, Address, City, State

I have implemented IEquatable on this CustObj so that it only compares against Address, City, and State, in the hopes of doing easy comparisons between the two lists.

The ultimate goal is to get the address ID from the database and update the address IDs for each address in the list of objects from the file. These two lists could have quite a lot of objects (over 1,000,000) so I want it to be fast.

The alternative is to offload this to the database and have the DB return the info we need. If that would be significantly faster/more resource efficient, I will go that route, but I want to see if it can be done quickly and efficiently in code first.

Anyways, I see there's a Zip method. I was wondering if I could use that to say "if there's a match between the two lists, keep the data in list 1 but update the address id property of each object in list 1 to the address Id from list 2".

Is that possible?

Answer 1

The answer is, it really depends. There are a lot of parameters you haven't mentioned.

The only way to be sure is to build one solution (preferably using the zip method, since it has less work involved) and if it works within the parameters of your requirements (time or any other parameter, memory footprint?), you can stop there.

Otherwise you have to off load it to the database. Mind you, you would have to hold the 1 million records from files and 1 million records from DB in memory at the same time if you want to use the zip method.

The problem with pushing everything to the database is, inserting that many records is resource (time, space etc) consuming. Moreover if you want to do that maybe everyday, it is going to be more difficult, resource wise.

Your question didn't say if this was going to be a one time thing or a daily event in a production environment. Even that is going to make a difference in which approach to choose.

To repeat, you would have to try different approaches to see which will work best for you based on your requirements: is this a one time thing? How much resources does the process have? How much time does it have? and possibly many more.

Answer 2

It kindof sounds also like a job for .Aggregate() aka

var aggreg = list1.Aggregate(otherListPrefilled, (acc,elemFrom1) => 
{  
     // some code to create the joined data, usig elemFrom1 to find
     // and modify the correct element in otherListPrefilled
    return acc;
});

normally I would use an empty otherListPrefilled, not sure how it performs on 100k data items though.

If its a onetime thing, its probably faster to put your file to a csv, import it in your database as temporary table and join the data in sql.