[英]How to import a big csv file into database with C# and Entity Framework?
I am using a basic streamreader to loop through a csv file of about 65gb (450 million rows).我正在使用一个基本的流阅读器来遍历一个大约 65gb(4.5 亿行)的 csv 文件。
using (sr = new StreamReader(currentFileName))
{
string headerLine = sr.ReadLine(); // skip the headers
while ((string currentTick = sr.ReadLine()) != null)
{
string[] tickValue = currentTick.Split(',');
// Ticks are formatted and added to the array in order to insert them afterwards.
}
}
This creates a list that will hold the ticks that belong to a candle and than call the insertTickBatch function.这将创建一个列表,该列表将保存属于蜡烛的报价,然后调用 insertTickBatch function。
private async static Task insertTickBatch(List<Tick> ticks)
{
if (ticks != null && ticks.Any())
{
using (DatabaseEntities db = new DatabaseEntities())
{
db.Configuration.LazyLoadingEnabled = false;
int currentCandleId = ticks.First().CandleId;
var candle = db.Candles.Where(c => c.Id == currentCandleId).FirstOrDefault();
foreach (var curTick in ticks)
{
candle.Ticks.Add(curTick);
}
await db.SaveChangesAsync();
db.Dispose();
Thread.Sleep(10);
}
}
}
This however takes about 15 years to complete and my intention is to speed this up.然而,这需要大约 15 年的时间才能完成,我的意图是加快速度。 How do I achieve this?
我如何实现这一目标?
I am not sure which EF you are using, but if available try this instead of your foreach loop:我不确定您使用的是哪个 EF,但如果可用,请尝试使用此方法代替您的 foreach 循环:
db.Ticks.AddRange(ticks);
Also, CSVHelper is a nice package that can convert your entire file into an Tick object list, and of course the Thread.Sleep has to go.此外,CSVHelper 是一个不错的 package,它可以将您的整个文件转换为 Tick object 列表,当然 Thread.Sleep 必须为 go。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.