LINQ - 通过WHERE子句查询大约6000条唯一记录

Question

I've got a problem and no idea how to solve it. Imagine you have a List<int> with about 6000 unique id's that matches the id's of a table with about a million records in a sql datatable. I want to select those 6000 records that matches those id's via LINQ from my c# program. I don't want to use Contains() because it gets very slow in translation and the argumentlist gets to big.

Any other ideas how to solve this ?

---

Something about my scenario ( this is not the real but a similar scenario ) :

I've got a Service that is connected to a database. A Client requests a batch of Items like a Person for example. The Service accepts the requests, query the DB and sends the data back to the client.

Person = (PersonID , Prename, Lastname)

Now the Client holds a temporary List of Persons. With an additional method I would like to retrieve the adresses of these Persons from the service. So I put a List of PersonID's to the service that should give me back a list of Adresses that have references to those persons.

Answer 1

I wouldn't recommend this. As great a tool as LINQ is, there are some scenarios where trying to get smart with your data by handling it code-side could be quite detrimental to the application performance.

You've got a list of these Id's somewhere, if they are in the database, why not do the whole operation as a stored procedure and just return the results, That way you're not having to push an expensive query across the wire, it's all in your database, so you minimise traffic, and likely increase responsiveness.

6000 items might not seem like a lot to bother with this for, but realistically as you said, it can be a bit of a nightmare of performance when trying to do a select with datasets of size.

Answer 2

Some ideas:

Insert the 6000 IDs into a temporary table and join that temporary table to your million record one.

Use Contains() and select in batches of n, where N = 500, 1000, etc. instead of all 6000 at once.

Using Contains() is going to make linq to create a very big SQL statement.

Answer 3

If you are using Entity Framework (EF) you can use inner join between your data (data) and your table (Customers)

void Main()
{    
    var data = Enumerable.Range(1, 6000);

    var result = from x in data
    join y in Customers
    on x equals y.CustomerID
    select x;

    result.Dump();
}

Answer 4

Typically, I've found that xml performs best for large IN criteria for IDs. It also gets around the 2100 max parameters in SQL Server, which you'll hit if you do a Contains in LINQ.

I would suggest:

• make List
• serialize to Xml
• create a stored procedure called ContainsXYZ that takes the xml as a parameter
• have your stored procedure use xpath to extract the Ids and join on it
• assuming you're using Entity Framework, you can map this stored procedure, execute it, then materialize the results into regular entities.

Answer 5

Just to get this out of the way...

 var joinTry = from company in dc.Companies
               join id in list on company.CompanyID equals id
               select company;

Doesn't work. LinqToSql won't let you join. "Local sequence cannot be used in LINQ to SQL implementations of query operators except the Contains operator."

 var containsTry = from company in dc.Companies
                   where list.Contains(company.CompanyID)
                   select company;

Does work. In the predictable

SELECT [t0].[CompanyID], [t0].[CompanyName]
FROM [Company] AS [t0]
WHERE [t0].[CompanyID] IN (@p0, @p1, @p2, @p3, @p4, @p5, ...

way... As dirty as this is, there is seriously no faster way to get a list of ints to a SQL server. The overhead time of any call is seriously larger than any parsing.

SELECT
    c.CompanyId,
    c.CompanyName
FROM Company c
    WHERE CompanyID IN (1,2,3,4,5,6,7,8,9,10)

...is the same execution speed as... (Generated by LINQ)

exec sp_executesql N'SELECT [t0].[CompanyID], [t0].[CompanyName]
FROM [Company] AS [t0]
WHERE [t0].[CompanyID] IN (@p0, @p1, @p2, @p3, @p4, @p5, @p6, @p7, @p8, @p9)',N'@p0 int,@p1 int,@p2 int,@p3 int,@p4 int,@p5 int,@p6 int,@p7 int,@p8 int,@p9 int',@p0=1,@p1=2,@p2=3,@p3=4,@p4=5,@p5=6,@p6=7,@p7=8,@p8=9,@p9=10

...and is Twice as fast as...

SELECT
    c.CompanyId,
    c.CompanyName
FROM Company c
        /* @Test is a table variable with 1-10 in it */
    INNER JOIN @Test t ON t.ID = c.CompanyID

You really don't need to optimize SQL server's handling of a list of integers. In the IN() solution, SQL puts the integers in an index it generates on the fly ANYWAY.

The real question should be... "What and I representing with a list of 6000 integers?" and "Should I put this list in a table?". Any solution that takes client-side list of 6000 integers, and sends it to the server, will be >= in overhead than a solution that uses Contains(). If you use LinqToSQL, you sort of have to sell out to the paradigm to a certain extent.

If this still makes you feel dirty, You might try creating a table for arbitrary restriction lists. Two columns, both ints. Then you can insert your IDs into that table, then just use...

var searchTry = from company in dc.Companies
                join search in dc.SearchLists on company.CompanyID equals search.ValueID
                where search.SearchID == savedSearchID
                select company;