使用EF多实体保存或更新数据

Question

I have a big problem when I was doing my code which involve insert data, or update if this exists, from a file. I'm using entity framework. files are near to 16 000 rows

I tried to insert them but I spent a lot of time doing this: more than 5 hour x file.

I decided to change it but i dont know how to start.

I read something about save each 1000, 5000, 10000 row, but I got and exception when i try to do it.

Store update, insert, or delete statement affected an unexpected number of rows (0).
Entities may have been modified or deleted since entities were loaded. Refresh
ObjectStateManager entries.

and when I tried to fix it I got this exception

AcceptChanges can not continue because the object's key values ​​conflict with another
object in ObjectStateManager. Make sure that the key values ​​are unique before calling 
AcceptChanges.

this is my code

//before this I passed db data to lists
foreach (var record in records)
{
    if (record != null)
    {
        try
        {
            if (!_location.Any(s => s.Id.Trim() == record.Id))
            {               
                Location l = new Location
                    {
                        Id = record.Id,
                        Name = record.Name,
                        Address = record.Address,
                        City = record.City,
                        State = record.State,
                        Zip = record.Zip.ToString(),
                        Zip2 = record.Zip2,
                        Custom1 = record.Custom1,                       
                        CreatedDate = date,
                        UpdatedDate = date
                    };

                db.location.Add(l);  
                db.SaveChanges();                              
                _location.Add(l);                                        
            }
            else
            {                                
                if (!_sales.Any(s => s.LocationId == record.Id && s.ReportDate == record.ReportDate))
                {
                    Sale s = new Sale
                    {
                        ReportDate = record.ReportDate,
                        CreatedDate = date,
                        UpdatedDate = date,
                        Amount = record.Amount,
                        LocationId = record.Id,
                    };
                    db.Sale.Add(s);  
                    db.SaveChanges();                  
                    _sales.Add(s);
                }
                else
                {

                    if (!_sales.Any(s => s.LocationId == record.Id && s.ReportDate == record.ReportDate && s.Amount == record.Amount))
                    {                       
                        if ((_sales.Where(s => s.LocationId == record.Id && s.ReportDate == record.ReportDate)).Count() == 1)
                        {                           
                            var sale = _sales.SingleOrDefault(s => s.LocationId == record.Id && s.ReportDate == record.ReportDate);
                            sale.UpdatedDate = date;
                            sale.Amount = record.Amount;
                            db.Entry(sale).State = EntityState.Modified;  
                            db.SaveChanges();                                                                   
                        }                                                
                    }
                }                                        
            }                        
        }
        catch (Exception ex)
        {

        }

    }
}

The changes I've done was:

change all db.SaveChanges for only one at the end of the foreach, and I got the First exception (also when I group them by 3000)

change the entity state, and I got the second exception.

I'llappreciate your help,

thanks.

Answer 1

As far as I understand, you want to import data from file to data. If I'a right, I was facing the same Problem, but with over 20,000 items per update. After some test I came to following solution which may solves your problem:

• the EF is much to slow for mass data Import
• so I used the SqlBulkCopy-Command to import the data in an extra import table and proccessed the imported data inside the database by stored procedures. A method to do Import the data could look like this:

private void DoMassDataImport(DataTable tab)
 {            
    var conn = new SqlConnection(this.sqlConnectionString);
    try
    {
        if (conn.State != ConnectionState.Open)
        {
            conn.Open();
        }

        SqlTransaction transaction = conn.BeginTransaction();

        using (SqlBulkCopy copy = new SqlBulkCopy(conn, SqlBulkCopyOptions.Default, transaction))
        {
            copy.BulkCopyTimeout = 10000;
            copy.DestinationTableName = "dbo.TargetTable";
            copy.WriteToServer(tab);
        }
        transaction.Commit();
        result.Success = true;                
    }
    catch (Exception ex)
    {
        result.Success = false;
        this.LogEvent(ex.Message + Environment.NewLine + ex.StackTrace);
    }
    finally
    {
        conn.Close();                
    }
}

• the stored procedures can be called from code, if the processing does not take longer than 1 Minute you could use the EF, but if it takes longer you should consider to use plain SQL commands like this:

private void ProcessImportedData()
{
    SqlConnection connection = null;    
    try
    {
        connection = new SqlConnection(this.sqlConnectionString);
        if (connection.State != ConnectionState.Open)
        {
            connection.Open();
        }

        SqlTransaction transaction = connection.BeginTransaction();
        using (SqlCommand processingCommand = connection.CreateCommand())
        {
            processingCommand.Transaction = transaction;
            processingCommand.CommandTimeout = 10000;
            processingCommand.CommandText = "dbo.StoredProcedure ";
            processingCommand.CommandType = CommandType.StoredProcedure;
            processingCommand.ExecuteNonQuery();            
            result.Success = true
        }

        transaction.Commit();
    }
    catch (Exception ex)
    {
        result.Success = false;     
    }
    finally
    {
        if (connection != null)
        {
            connection.Close();

        }
    }
}

• using this approach reduced the time for the data Import from more than 5 hours to less than 5 minutes