使用LINQ将大量数据插入数据库

Question

I want to insert around 1 million records into a database using Linq in ASP.NET MVC. But when I try the following code it didn't work. It's throwing an OutOfMemoryException . And also it took 3 days in the loop. Can anyone please help me on this???

db.Database.ExecuteSqlCommand("DELETE From [HotelServices]");

DataTable tblRepeatService = new DataTable();
tblRepeatService.Columns.Add("HotelCode",typeof(System.String));
tblRepeatService.Columns.Add("Service",typeof(System.String));
tblRepeatService.Columns.Add("Category",typeof(System.String));

foreach (DataRow row in xmltable.Rows)
{
     string[] servicesarr = Regex.Split(row["PAmenities"].ToString(), ";");

     for (int a = 0; a < servicesarr.Length; a++)
     {
         tblRepeatService.Rows.Add(row["HotelCode"].ToString(), servicesarr[a], "PA");
     }

     String[] servicesarrA = Regex.Split(row["RAmenities"].ToString(), ";");

     for (int b = 0; b < servicesarrA.Length; b++)
     {
         tblRepeatService.Rows.Add(row["hotelcode"].ToString(), servicesarrA[b], "RA");
     }
}

HotelAmenties _hotelamenties;

foreach (DataRow hadr in tblRepeatService.Rows)
{
     _hotelamenties = new HotelAmenties();
     _hotelamenties.Id = Guid.NewGuid();
     _hotelamenties.ServiceName = hadr["Service"].ToString();
     _hotelamenties.HotelCode = hadr["HotelCode"].ToString();

     db.HotelAmenties.Add(_hotelamenties);
}

db.SaveChanges();

tblRepeatService table has around 1 million rows.

Answer 1

Bulk inserts like this are highly inefficient in LINQtoSQL. Every insert creates at least three objects (the DataRow , the HotelAmenities object and the tracking record for it), chewing up memory on objects you don't need.

Given that you already have a DataTable , you can use System.Data.SqlClient.SqlBulkCopy to push the content of the table to a temporary table on the SQL server, then use a single insert statement to load the data into its final destination. This is the fastest way I have found so far to move many thousands of records from memory to SQL.

Answer 2

If performance doesn't matter and this is a 1 shot job you can stick to the way you're using. Your problem is you're only saving at the end, so entity Framework has to store and generate the SQL for 1 million operations at once, modify your code so that you save every 1000 or so inserts instead of only at the end and it should work just fine.

int i = 0;
foreach (DataRow hadr in tblRepeatService.Rows)
{ 
     _hotelamenties = new HotelAmenties();
     _hotelamenties.Id = Guid.NewGuid();
     _hotelamenties.ServiceName = hadr["Service"].ToString();
     _hotelamenties.HotelCode = hadr["HotelCode"].ToString();

     db.HotelAmenties.Add(_hotelamenties);
     if((i%1000)==0){
     db.SaveChanges();
     }
     i++;
}    
db.SaveChanges();