LINQ to SQL: how to select specific fields conditionally into DTOs (performance)

Question

LINQ to SQL is great however it backfires in performance in some cases. And with my case it backfires bigtime.

I have a StudentServiceDm that maps to a StudentService Table with 40 some fields. I have 10-20 millions of services in the table. The requirement from the client is that they need to export all services into excel from the application, while selecting only the fields they want at the time.

From my understanding, if i do

IQueryable<StudentServiceDm> serviceQuery = GetListQuery();
List<StudentServiceDto> dtos = serviceQuery.Select(m => new StudentServiceDto
{
    Id = m.Id,
    Name = m.Name,
}).ToList();

Then this translates into the SQL that selects only the fields I need

SELECT Id, Name FROM StudentService

Which saves performance by alot by decreasing the number of fields that need to be selected.

But in my case I need to allow the user to conditionally select fields like this, so I do this:

IQueryable<StudentServiceDm> serviceQuery = GetListQuery();

List<StudentServiceDto> dtos = serviceQuery.Select(m => new StudentServiceDto
            {
                Id = (options.ShowId) ? m.Id : null,
                Name = (options.ShowName) ? m.Name,
                StudentFirstName = (options.ShowFirstName) ? m.Student.FirstName : null,
                StudentLastName = (options.ShowLastName) ? m.Student.LastName : null,
                StudentServiceType = (options.ShowType) ? m.StudentServiceType.Name : null,
                StudentServiceSubType = (options.ShowSubType) ? m.StudentServiceSubType.Name : null,
                Date = (options.ShowDate) ? m.Date : null,
                // alot more assignments .....
            }).ToList();

However, this translates into the SQL query to something like this (according to SQL Profiler:

SELECT 
    CASE WHEN (@p__linq__1 = 1) THEN [Project2].[Date] END AS [C1], 
    CASE WHEN (@p__linq__2 = 1) THEN [Project2].[Name] END AS [C2], 
    CASE WHEN (@p__linq__3 = 1) THEN [Project2].[StudentServiceTypeId] END AS [C3], 
    CASE WHEN (@p__linq__4 = 1) THEN [Project2].[StudentServiceSubTypeId] END AS [C4], 
    CASE WHEN (@p__linq__5 = 1) THEN [Project2].[Date] END AS [C5], 
    // ..... more fields
    FROM * // 

where it puts a condition on every field to check if it should retrieve ... and actually this query's performance is almost the same as if all fields were selected.

What I want to achieve is, if the user only chooses to select Name and Date , then the translated SQL should only be

SELECT Name, Date FROM StudentService

After some performance analysis, the performance difference between these 2 sets of statement is about 3-4 times.

Any expert opinion?

Answer 1

You can create Linq expressions at runtime that exactly correspond to what you'd type in inside a .Select() statement, it's just not very straightforward. I never did it for Linq to SQL specifically but I don't see why it wouldn't work. Something like How do I dynamically create an Expression<Func<MyClass, bool>> predicate? but of course I'd recommend reading the docs first to get familiar with expressions.

Answer 2

The only way to do what you are interested in doing is to create lots of different Select() sections, each corresponding to a different combination of fields. However, you will need to create an awful lot of different combinations (N!, where N is the number of fields), so it is not really practical or maintainable.

There is no other way to do this using L2S.

If performance is important for you in this aspect, I suggest creating the sql directly and using something like Dapper to retrieve the data. Then you can easily control exactly the fields that you want to include.