如何用两个相关表优化一个简单的LINQ查询？

Question

I have two related tables with the following structure:

'patients' :

{ Id = 1, Surname = Smith998 }
...
{ Id = 1000, Surname = Smith1000 }

and the second is 'receptions' :

{ PatientId = 1, ReceptionStart = 3/3/2017 1:14:00 AM }
{ PatientId = 1, ReceptionStart = 1/7/2016 1:14:00 AM }
...
{ PatientId = 1000, ReceptionStart = 1/23/2017 1:14:00 AM }

the tables are not from a database, but they are generated with the following sample code:

        var rand = new Random();
        var receptions = Enumerable.Range(1, 1000).SelectMany(pid => Enumerable.Range(1, rand.Next(0, 10)).Select(rid => new { PatientId = pid, ReceptionStart = DateTime.Now.AddDays(-rand.Next(1, 500)) })).ToList();
        var patients = Enumerable.Range(1, 1000).Select(pid => new { Id = pid, Surname = string.Format("Smith{0}", pid) }).ToList();

The question is what is the optimal way to select the patients that have receptions before 1/1/2017?

Of cause I can write something like this:

        var cured_receptions = (from r in receptions where r.ReceptionStart < new DateTime(2017, 7, 1) select r.PatientId).Distinct();
        var cured_patients = from p in patients where cured_receptions.Contains(p.Id) select p;

but it is not clear for me what 'cured_receptions.Contains(p.Id)' code actually does? Does it simply iterate over all the patients searching the Id or it use something like indices in a database? Can cured_receptions.ToDictionary() or something like this help here somehow?

Answer 1

but it is not clear for me what 'cured_receptions.Contains(p.Id)' code actually does? Does it simply iterate over all the patients searching the Id or it use something like indices in a database?

Case 1: Interacting with Database

If you were interacting with the database then no query will be sent to the database until you execute the 2nd query by calling ToList() on it or by iterating the items in cured_patients . The query sent to the database will be something along the lines:

SELECT 
[Extent1].[Id] AS [Id], 
[Extent1].[Surname] AS [Surname]
FROM [dbo].[Patients] AS [Extent1]
WHERE  EXISTS (SELECT 
    1 AS [C1]
    FROM [dbo].[Receptions] AS [Extent2]
    WHERE ([Extent2].[ReceptionStart] < 
    convert(datetime2, '2017-07-01 00:00:00.0000000', 121)) 
    AND ([Extent2].[PatientId] = [Extent1].[Id])
)

Will it use any indices?

Yes if PatientId , Id and ReceptionStart are indexed, then the database server will use them.

Case 2: Interacting with items in memory

For the first query it will iterate all receptions , find the ones whose ReceptionStart is before the given date, select the PatientId and then remove any duplicate PatientId (s).

Then the 2nd query which is below:

var cured_patients = 
   from p in patients 
   where cured_receptions.Contains(p.Id) 
   select p;

Will iterate each item in patients and see if the Id of that item is found in cured_receptions . For all the items whose Id is found in the cured_receptions , it will select those items. Contains simply returns a true or false .

Answer 2

Starting over assuming everything in memory only...

1. Your cured_receptions isn't evaluated until called by the Contains so it would be much more efficient use put .ToList() on the end of that variable definition (about 100X faster).

2. LINQ doesn't "search" - Contains is doing the searching. If you want to use something like a binary search or better still, a hash table, you must create it. If you do use a HashSet<int> then you will gain another 47X speedup. Taking the Distinct off (since the HashSet will handle that) saves another 15%.

3. Remembering your constants in variables instead of creating them as you go ( new DateTime ...) may save a little more. Even greatly increasing your random data doesn't take up enough time to tell with the HashSet .

4. Using a join is faster than your initial query, but your query combined with a HashSet is fastest.

So the fastest code is:

var cured_receptions = new HashSet<int>((from r in receptions where r.ReceptionStart < endDateTime select r.PatientId));
var cured_patients = from p in patients where cured_receptions.Contains(p.Id) select p;

Note: I used LINQPad to generate timings and sample data. I changed your date parameters because your values made the majority of receptions match.

Here is the code from my LINQPad:

var rand = new Random();
var begin = DateTime.Now;
var receptions = Enumerable.Range(1, 100000).SelectMany(pid => Enumerable.Range(1, rand.Next(0, 100)).Select(rid => new { PatientId = pid, ReceptionStart = begin.AddDays(-rand.Next(1, 180)) })).ToList();
var patients = Enumerable.Range(1, 100000).Select(pid => new { Id = pid, Surname = string.Format("Smith{0}", pid) }).ToList();

var startTime = Util.ElapsedTime;
var endDateTime = new DateTime(2017, 5, 1);
//var cured_receptions = (from r in receptions where r.ReceptionStart < new DateTime(2017, 5, 1) select r.PatientId).Distinct().ToList();
//var cured_receptions = (from r in receptions where r.ReceptionStart < new DateTime(2017, 5, 1) select r.PatientId).Distinct();
//var cured_receptions = new HashSet<int>((from r in receptions where r.ReceptionStart < new DateTime(2017, 5, 1) select r.PatientId).Distinct());
//var cured_receptions = new HashSet<int>((from r in receptions where r.ReceptionStart < endDateTime select r.PatientId).Distinct());
//var cured_receptions = new HashSet<int>((from r in receptions where r.ReceptionStart < new DateTime(2017, 5, 1) select r.PatientId));
var cured_receptions = new HashSet<int>((from r in receptions where r.ReceptionStart < endDateTime select r.PatientId));
var cured_patients = from p in patients where cured_receptions.Contains(p.Id) select p;

//  var cured_patients = (from r in receptions
//                       where r.ReceptionStart < endDateTime
//                       join p in patients on r.PatientId equals p.Id
//                       select p).Distinct();

//  var cured_patients = from p in patients
//                       join r in receptions on p.Id equals r.PatientId into rj
//                       where rj.Any(r => r.ReceptionStart < endDateTime)
//                       select p;

cured_patients.Count().Dump();
var endTime = Util.ElapsedTime;

(endTime - startTime).Dump("Elapsed");