在LINQ to Entities中检查重复项的最快方法是什么？

Question

I have a table that is storing strings in a SQL Azure table. The user can upload files of new strings from a web browser, and I am checking for duplicates based on the Entity Framework context. My code to add the de-duplicated strings to the context looks like this:

using (StreamReader sr = new StreamReader(theStream))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {
        if (!context.MyEntity.Any(o => o.String == line))
        {
            theString = new DAL.TheString();
            theString .String = line;
            context.MyEntity.Add(theString );
            totalAdded++;
        }
    }
}

Using MyEntity.Any() is way, way too slow. Handling 20,000 strings takes 40 minutes, and some simple orchestration seems to point to the duplicate check.

My question is: what is the fastest way to do this within EF? Is L2E not the best tool for the job here? Should I get rid of EF altogether? Or should I just queue up the files and set up a background worker, 'cause this is ALWAYS going to be slow.

Answer 1

Assuming your database table isn't so large that the strings cannot all fit into memory, you can put them into a HashSet through one query and then query against that in-memory collection:

var lines = new HashSet<string>(context.MyEntity.Select(o => o.Property));
using (StreamReader sr = new StreamReader(theStream))
{
    while (!sr.EndOfStream)
    {
        string line = sr.ReadLine();
        if (lines.Add(line))
        {
            //add line
        }
    }
}

If you don't have enough memory for this to work, then your best bet would likely be to create a new trigger in the database to verify that the property is unique, and that will throw out records that attempt to create duplicates. Then you can attempt to add all of the lines from your stream and let the DB sort out which ones to keep when it gets them all.