比较csv中的记录和大列表

Question

In my website user uploads a csv file.

I am reading a csv file using this library http://www.codeproject.com/Articles/11698/A-Portable-and-Efficient-Generic-Parser-for-Flat-F The csv file will have around 4000 records (each record with 5 columns).

I am reading each record in to a List and search in a large list of objects(Before started reading csv file I am reading the large list of objects from a service to the cache.) to check whether this record already exist or not.

In this way I have to do 4000 iterations and in each iteration I have to search in large list of objects ( around 100 thousands records which are in cache).

Is this the good way of implementation? Is there any way to improve the speed? Is it good idea to store such a large list in cache?

My environment is VS2010, .NET4.0,

Answer 1

You can speed up your search by using an appropriate data structure for your lists. If the items have a unique/primary key you could use a hashmap which would be more efficient than iterating the whole list for each item. This way you could use hashmap.containskey().

if you run the service you could push the responsibility up to the service - perhaps pushing the list of unique keys there for comparison.

maybe you could post some code for a more specific answer.