What is the best way in LINQ to sort 1000000 Records

Question

I want to sort and page 1,000,000 records with Linq. I don't know if the way I use to get the data or correct or not because the pages goes so slowly.

Here is my code:

public HttpResponseMessage GetAllProducts(int page, string SortColumn,string Name = null)
{
    const int PageSize = 4;
    HttpResponseMessage response = null;
    IEnumerable<Product> result = null;

    if (string.IsNullOrEmpty(Name))
    {
        result = db.Products.OrderBy(SortColumn).AsEnumerable();

    }
    else
    {
        result = db.Products
            .Where(p => p.Name.StartsWith(Name))
            .OrderBy(SortColumn).AsEnumerable();
    }


    int NumberOfPages = result.Count();
    var begin = (page - 1) * PageSize;
    var data = result.Skip(begin).Take(PageSize).AsEnumerable();


    ProductPager myproduct = new ProductPager
    {
        ProductList = data,
        TotalRecords = NumberOfPages

    };
    response = Request.CreateResponse(HttpStatusCode.OK, myproduct);
    return response;


}

Answer 1

You are currently pulling all 1 million records out of your database into memory, and applying your Skip() and Take() to that collection. This is very expensive. Change your IEnumerable<Product> into an IQueryable<Product> and get rid of the calls to .AsEnumerable() .

Here's what I'd do:

public HttpResponseMessage GetAllProducts(int page, string sortColumn, string name = null)
{
    const int PageSize = 4;
    IQueryable<Product> query = db.Products;

    if (!string.IsNullOrEmpty(Name))
    {
        query = query.Where(p => p.Name.StartsWith(name));
    }

    int numberOfRecords = result.Count();
    var begin = (page - 1) * PageSize;
    var data = query.OrderBy(sortColumn)
        .Skip(begin).Take(PageSize)
        .ToList();

    ProductPager myproduct = new ProductPager
    {
        ProductList = data,
        TotalRecords = numberOfRecords 
    };
    return Request.CreateResponse(HttpStatusCode.OK, myproduct);
}

What's Happening?

Entity Framework is a LINQ Query Provider . When you access db.Products , that's returning an object which implements IQueryable<Product> and IEnumerable<Product> . This gives you two sets of LINQ extension methods, many of which overlap each other (eg Where() , Skip() , Take() , OrderBy() , and Count() ).

Calling the methods that pertain to IQueryable<> , will do one of two things:

1. For operations that don't require immediate evaluation (like Where() , and OrderBy() ) no actual work is done in relation to the database: you just get another IQueryable<> that records the fact that you wanted to call a particular LINQ method with specific parameters.
2. For operations that require immediate evaluation (like Count() ), a SQL Query will be issued representing the query that you've built up so far, and you'll retrieve the result that is needed. For example, SQL Server will actually count the necessary records, and return just a number, instead of returning individual records.

On the other hand, if you call the methods that pertain to IEnumerable<> , you produce an object that will (either immediately or when evaluated later) perform the original query (giving you all Products in the database) and then iterate over it to do things like filtering, skipping, taking, ordering, and counting.

Since IQueryable<> is more specific than IEnumerable<> , the IQueryable<> extension methods are typically invoked, unless you go out of your way to cast the result as an IEnumerable<> (which is what you've done in your code).