Working with a large amount of data

Question

I got c# appliaction and entity famework as ORM. I got database with table Images . Table have Id , TimeStamp , Data columns. This table can have really ALOT entities. Also Data column contain large byte array. I need to take first entity starting from some date, or first 5 as example.

var result = Images.OrderBy(img => img.TimeStamp).FirstOrDefault(img => img.TimeStamp > someDate);

throws out of memory exception. Is there some way to pass that? Should i use stored procedure or something else?

Answer 1

If Images is already a queried object, then when you OrderBy it, it accesses the whole set. I'll assume it isn't, and it is directly your DbSet or an EF IQueryable (so you are querying using Linq-To-Entities and not Linq-To-Objects and the ordering is done on the query to the database, and not on the returned whole set).

Unless you need change tracking detection, use AsNoTracking on your DbSet (in this case, Context.Images.AsNoTracking().OrderBy(...) . That should lower the memory requirements by a lot (change tracking detection requires more than twice the memory).

Also, if using large blob data, it might be wise to store it in its own table (with just an id and the data ) and access it only when you need it (having a reference to this id on the table/entity where you are doing your operations) if you are using an ORM and want to work with the original entity all the time (you could also use a Select to project the query on a new entity without the blob field).

If you need to access the image data for the returned rows all the time, and there's not enough memory in the system for it, then tough luck.