How to import a big csv file into database with C# and Entity Framework?

Question

I am using a basic streamreader to loop through a csv file of about 65gb (450 million rows).

using (sr = new StreamReader(currentFileName))
{
    string headerLine = sr.ReadLine(); // skip the headers

    while ((string currentTick = sr.ReadLine()) != null)
    {
        string[] tickValue = currentTick.Split(',');
        // Ticks are formatted and added to the array in order to insert them afterwards.
    }
}

This creates a list that will hold the ticks that belong to a candle and than call the insertTickBatch function.

    private async static Task insertTickBatch(List<Tick> ticks)
    {
        if (ticks != null && ticks.Any())
        {
            using (DatabaseEntities db = new DatabaseEntities())
            {
                db.Configuration.LazyLoadingEnabled = false;
                int currentCandleId = ticks.First().CandleId;
                var candle = db.Candles.Where(c => c.Id == currentCandleId).FirstOrDefault();

                foreach (var curTick in ticks)
                {
                    candle.Ticks.Add(curTick);
                }

                await db.SaveChangesAsync();
                db.Dispose();
                Thread.Sleep(10);
            }
        }
    }

This however takes about 15 years to complete and my intention is to speed this up. How do I achieve this?

Answer 1

I am not sure which EF you are using, but if available try this instead of your foreach loop:

 db.Ticks.AddRange(ticks);
            

Also, CSVHelper is a nice package that can convert your entire file into an Tick object list, and of course the Thread.Sleep has to go.