搜索1GB CSV文件

Question

I have a CSV file. Each line is made up of the same format eg/

I,h,q,q,3,A,5,Q,3,[,5,Q,8,c,3,N,3,E,4,F,4,g,4,I,V,9000,0000001-100,G9999999990001800000000000001,G9999999990000001100PDNELKKMMCNELRQNWJ010, , , , , , ,D,Z,

I have a Dictionary<string, List<char>>

It is populated by opening the file, reading each line, taking elements from the line and adding it to the dictionary, then the file is closed.

The dictionary is used elsewhere in the program where it accepts input data into the program and then finds the key in the dictionary and uses the 24 elements to compare against the input data.

StreamReader s = File.OpenText(file);
 string lineData = null;
 while ((lineData = s.ReadLine()) != null)
 {
   var elements = lineData.Split(',');
   //Do stuff with elements
   var compareElements = elements.Take(24).Select(x => x[0]);
   FileData.Add(elements[27], new List<char>(compareElements));

  }
  s.Close();

I have just been told that the CSV file will now be 800mb and have roughly 8 million records in it. I have just tried to load this up on my Dual Core Win 32bit laptop with 4GB of RAM in debug and it threw a OutOfMemoryException .

I am now thinking that not loading the file into memory will be the best bet but need to find a way to search the file quickly to see if the input data has a matching item equal to element[27] and then take the first 24 elements in that CSV and compare it to the input data.

a) Even if I stuck with this approach and used 16GB RAM and Windows 64bit would having that many items in a dictionary be ok?

b) Could you provide some code/links to ways to search a CSV file quickly if you dont think using a dictionary is a good plan

UPDATE: Although I have accepted an answer, I just wondered what people's thoughts were on using FileStream to do a lookup and then extract data.

Answer 1

如果您打算搜索这么多记录，我建议将文件批量插入到SQL Server这样的DBMS中，并为其指定适当的字段索引，然后使用SQL查询来检查记录是否存在。

Answer 2

• forget MS access. Really.
• try sqlite, it will be more than adequate for a few million rows
• if you can't index your data, then don't use a database, use an external utility like egrep with an appropriate regex to search for specific fields. It will be much faster.

Answer 3

We had a similar problem with importing a large csv file containing data that needed to be aggregated. In the end we did a bulk insert into a SQL Server table and used SQL to perform the aggregation. It was pretty quick in the end (a couple of minutes end-to-end).

Answer 4

There are several options available to you but yes, I would agree that loading this data into memory is not the best option.

a) You could load the data into a relational database although this may be overkill for this type of data.

b) You could use a NoSQL solution like RavenDB . I think this may be a good option for you.

c) You could use a more efficient physical storage option like Lucene

d) You could use a more efficient in-memory/caching option like Redis .

Answer 5

一种解决方案是将文件分解为一些较小的文件，并在每个文件中进行并行搜索，搜索顺序将小于或等于n（读取整个文件）

Answer 6

As the rest of your program uses the StringDictionary entries, you still ideally need to store your results in memory - you dont really want to be querying out to a DB 1000's of times. (This could depend if your program lives on the DB server)!

I'd look into the memory usage of StringDictionary for your structure and see what your theoretical maximums are and see if you can cover this in a caveat on the functional requirements. Otherwise look for a more efficient way to store - streaming your results out to an XML file for example will be quicker to access than a DB.