At what point do I need to off-load the work of a query to the DB?

Question

I have a web app, and I'm connecting to a DB with entity framework.

To select all Employee records of a particular department, for example, I can quite easily write:

....Employees.Where(o => o.Department == "HR").ToList();

Works fine. But is it most optimal?

Should this Where clause be incorporated into a stored procedure or view? Or does my entity framework code do the job of converting it to SQL anyway?

We've had performance problems in our team in the past from when people pull records into memory and then do the filtering in .net instead of at a database level. I'm trying to avoid this happening again so want to be crystal clear on what I must avoid.

Answer 1

If Employees is provided by Entity Framework then the Where() will be translated into SQL and sent to the database. It is only the point that you materialise the objects does it take the filters you have applied before and turn them into SQL. Anything after that point is just plain LINQ to objects.

Methods that cause materialisation to happen include things like .ToList() and .ToArray() (there are more, but these two are probably the most common).

If you want to see what is happening on SQL Server, you should open up the SQL Profiler and have a look at the queries that are being sent.

Answer 2

We've had performance problems in our team in the past from when people pull records into memory and then do the filtering in .net instead of at a database level.

Just as an addendum to Colin's answer, and to target the quote above, the way to avoid this is to make sure your database queries are fully constructed with IQueryable<T> first, before enumerating the results with a call such as .ToList() , or .ToArray() .

As an example, consider the following:

IEnumerable<Employee> employees = context.Employees;
// other code, before executing the following
var hrEmployees = employees.Where(o => o.Department == "HR").ToList();

The .ToList() will enumerate the results grabbing all of the employees from the context first, and then performing the filtering on the client. This isn't going to perform very well if you've got a lot of employees to contend with, and it's certainly not going to scale very well.

Compare that with this:

IQueryable<Employee> employees = context.Employees;
// other code, before executing the following
var hrEmployees = employees.Where(o => o.Department == "HR").ToList();

IQueryable<T> derives from IEnumerable<T> . The difference between them is that IQueryable<T> has a query provider built in, and the query you construct is represented as an expression tree. That means it's not evaluated until a call that enumerates the results of the query, such as .ToList() .

In the second example above, the query provider will execute SQL to fetch only those employees that belong to the HR department, performing the filtering on the database itself.