简体   繁体   English

使用CSV文件作为数据库的效率

[英]Efficiency of using CSV file as database

I have a large (350 MB) CSV file that I need to query. 我有一个需要查询的大型(350 MB)CSV文件。 I get the data file as a data dump once every day, so I could do some daily processing to this data. 我每天都会以数据转储的形式获取数据文件,因此可以对这些数据进行一些日常处理。 Once I have the data, I want to query it at more frequent intervals. 有了数据后,我想更频繁地查询一次。 I'm wondering how best to do this. 我想知道如何最好地做到这一点。

I was thinking that I have two alternatives. 我当时以为我有两种选择。

  1. Connect with an OleDbConnection and then query directly using an OleDbDataAdapter , like this OleDbConnection连接,然后使用OleDbDataAdapter直接查询,如下所示

     OleDbConnection conn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + base.applicationRoot + ";" + "Extended Properties=\\"text;HDR=Yes;FMT=CSVDelimited\\""); OleDbDataAdapter adapter = new OleDbDataAdapter(String.Format("select something from [{0}]", base.csvFileName), conn); 
  2. Transfer the whole csv file, as I get it, into a 2012 sql server database table, and then query from that sql table instead. 当我得到它时,将整个csv文件传输到2012 sql服务器数据库表中,然后从该sql表中查询。

What is the better approach. 有什么更好的方法。 Or, how inefficient is it to use a CSV file as a database (read-only)? 或者,将CSV文件用作数据库的效率低下(只读)?

I would insert the csv into a database for more powerful query capabilites. 我会将csv插入数据库以获取更强大的查询功能。 If you're using SQL server you can even bulk insert the csv using a standard bulk insert command. 如果您使用的是SQL Server,甚至可以使用标准的批量插入命令批量插入csv。

http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/ http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-服务器/

If its an offline solution, you could use SQLite instead of Sql Server. 如果是脱机解决方案,则可以使用SQLite代替Sql Server。 That would give good performance compared to a CSV, but with the overhead of constructing it once a day. 与CSV相比,这将提供良好的性能,但是每天构造一次的开销很大。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM