C＃LINQ-SQL帮助-查询超过10,000个项目的数据库

Question

I'm running into an issue with LINQ -> SQL speed at the moment. I am trying to retrieve over 10,000 products - Which is fine as the speed to retrieve just the code,name,description,id for these to a list is around 2 seconds.

My main issue is then trying to iterate over these 10,000 items to calculate the count of stock (Based on how many stock items (rows) than are matched in the db table). This of course makes over 10,000 calls to SQL to work this out - which takes a lot of time. I need this calculation to then base my filter to only those with a particular amount.

 IEnumerable<Product> filteredRecords = warehousing.GetProducts(wID);  // Takes under 2 seconds for 10k records

                var data = filteredRecords
                                .Select(p => new
                                {
                                    ID = p.ProductID,
                                    ProductName = p.Product.Name,
                                    ProductCode = p.Product.Code,                                        
                                    StockLevel = stock.GetTotalStockCount(p.ProductID, p.WarehouseID.Value)
                                }).OrderBy(l => l.StockLevel).ToArray();

Should I go down the path of trying to revert to Native Query to slightly increase the speed or should I opt for a background process that calculates the stock counts periodically and stores them into a separate table?

I'm unsure of what is best practice/approach in this situation?

Edit It's this calculation that's adding the penalty. Including the GetTotalStockCount method:

        public int GetTotalStockCount(Int32? productID = null, Int32? warehouseID = null)
    {

        var query = from stockItem in _unitOfWork.StockItemRepository.Get(w => w.ProductID == productID).Select(s => s.LocationID)
                    join warehouseLocation in _unitOfWork.WarehouseLocationRepository.Get(w => w.WarehouseID == warehouseID && w.Type == WarehouseLocationType.Stock).Select(s => s.ID) on stockItem.Value equals warehouseLocation
                    select new { stockItem.Value };

        return query.Count();
    }

Answer 1

You should filter your data with StockCount and should not use ToArray until you really need.

Try this:

 IEnumerable<Product> filteredRecords = warehousing.GetProducts(wID);  // Takes under 2 seconds for 10k records

var data = filteredRecords
    .Select(p => new
    {
        ID = p.ProductID,
        ProductName = p.Product.Name,
        ProductCode = p.Product.Code,                                        
        StockLevel = stock.GetTotalStockCount(p.ProductID, p.WarehouseID.Value)
    }).Where(l => l.StockLevel > NEEDED_AMOUNT);

And consider move GetTotalStockCount method on database layer to filter your data there.

Answer 2

You have two options:

1. Probably you should consider to use view on database to generate this data, because GetTotalStockCount method cannot be translated to SQL query to speed up. Of course it depends on complexity of GetTotalStockCount method.

2. In first query get records from Products, StockItem and WarehouseLocation and then group them by Products to get count aggregate

Answer 3

You need to change method GetTotalStockCount to get count from a memory list instead of database.

So, first fetch and save the count from DB in a list and use that list inside function GetTotalStockCount.

Answer 4

The problem is that the GetTotalStockCount method is hitting the db in a separate, once for each record. I'm not sure I have your db structure right here but you could try this:

.Select(p => new
    {
        ID = p.ProductID,
        ProductName = p.Product.Name,
        ProductCode = p.Product.Code,                                        
        StockLevel = p.StockItem.WarehouseLocation.Where(w => w.WarehouseID == p.WarehouseID.Value).Count()
    }).OrderBy(l => l.StockLevel).ToArray();

This will calculate the stock level item within the sql query and mean that there is only one db hit.

NOTE: .StockITem.WarehouseLocation may change depending on how you db is structured but this is how I imagine it would look from you current query.

Answer 5

You could try the TSQL

select p.ProductID, p.Product.Name, p.Product.Code, count(*)    
  from StockItemRepository p    
  join WarehouseLocationRepository w
    on p.ProductID = w.WarehouseID.Value    
   and w.Type == WarehouseLocationType.Stock   
 group by p.ProductID, p.Product.Name,  p.Product.Code