C# Entity Framework - Optimization/Best Practices

Question

I have an entity framework database that has the following structure:

"Lots" are a named collection of parts

"Parts" are individual pieces

"Stations" are a result of Parts

"Tools" are a result of Stations

"Measurements" are a result of Tools

For a given lot with approx. 53,000 parts there are 6 associated rows in my test case for a total row count of 318,001 (+ 1 for a lot)

My question is how can I better the performance of Entity Framework with this type of data association.

To delete the lot (Using CASCADE delete with MySQL) takes approx. 12 seconds. To pull the lot data takes approx. 2 minutes

I am utilizing the following to pull Lot Information:

this.Context.Database.CommandTimeout = 2700;

Lot foundLot = EntitiesContext.Lots.Where(x => x.ID == lotID).FirstOrDefault();

this.Context.Database.CommandTimeout = 180;

I am utilizing the migrations in Entity Framework Code First

 Database.SetInitializer(new MigrateDatabaseToLatestVersion<GenInspContext, Migrations.Configuration>());

When I create my tables I set cascadeDelete to true

.ForeignKey("dbo.Lots", t => t.LotID, cascadeDelete: true)

I am more concerned about the time it takes to pull data than I am about the 12 seconds to remove data.

Answer 1

The first, and arguably biggest performance roadblock developers hit with Entity Framework is lazy loading. It's a great feature to ensure applications work rather than throw NullReferenceException s but it is an absolute performance killer when you don't account for it.

Statements like:

Lot foundLot = EntitiesContext.Lots.Where(x => x.ID == lotID).FirstOrDefault();

... look innocent enough, and they are completely harmless provided you only want information from the Lot record, not any other related entity's information. This includes doing something like serializing the Lot to send to a web client.

By default, whenever EF loads an entity that includes virtual references to other entities, unless you explicitly disable lazy loading proxies, EF will create proxies of the entities that will track those related entities. If the reference is not eager loaded and code attempts to access it, the entity will check to see if it's DbContext is still available and issue a new query to fetch the touched entity.

Combine this behaviour with a serializer and this leads to all KINDS of hell from a performance standpoint. For instance, if a Lot has a collection of Parts, and each part has a collection of Stations and we load 1 single Lot we get:

SELECT * FROM Lot WHERE LotId = 1;

Innocent enough. However, if our serializer or code touches the Parts collection, the following query gets run:

SELECT * FROM Parts WHERE LotId = 1;

Now that loads 100 parts... Still, nothing to be really concerned about. However, the more nested the object model is, the worse it quickly becomes. In the case of a serializer, it will start iterating over each part, which touches the Stations collection. Now the poop starts hitting the fan:

SELECT * FROM Stations WHERE PartId = 1;
SELECT * FROM Stations WHERE PartId = 2;
SELECT * FROM Stations WHERE PartId = 3;
SELECT * FROM Stations WHERE PartId = 4;
SELECT * FROM Stations WHERE PartId = 5;
....
SELECT * FROM Stations WHERE PartId = 100;

It's doing this (rather than JOIN ing Part on Station) because the iteration is touching each Part's Station collection. This results in a Stations query for each part. If each part has 1-5 stations. As those Stations get serialized, the references for each station in turn result in more queries, and so on. You can watch this proliferation of SQL spam using profilers while your application is running.

The quick fix is to leverage eager loading. This replaces the SELECT spam with JOIN s in the initial query. This can make loading the data many factors faster than relying on lazy loading. One query with the various inner and outer joins may take seconds compared to waiting minutes for EF to compose and kick off tens of thousands of queries. However, the consideration here is that you are still loading a potential crap-ton of data from the server. EF has to compose the query, send it to the DB to execute, the DB Server needs to allocate the space for all of the results, then transmit those over the wire back to the app server, which then has to allocate the memory for the result graph, potentially serializing that to the client. This also leaves landmines lurking in your application as it evolves and new relationships are added to existing entities. One new relationship gets added somewhere in the graph and before you know it there are seemingly unrelated performance issues popping up in completely untouched areas like searching because a serializer is now picking up and loading those entities which weren't eager loaded. (Since those areas of the app don't need to display that info.)

The better solution is to leverage projection whenever reading data to be computed against or serialized to a client, and save loading entities and their related details (if necessary) for things like Update operations. Projection with Select or Automapper's ProjectTo method allow you to compose queries that populate a safe, POCO to serialize or an anonymous type to inspect and consume which results in far, far, faster querying and much smaller payloads of data to send over the wire. This way the query only ever loads the information it needs, and changes to the data model in terms of new relationships never pollute existing queries. Those existing queries can remain oblivious to the new tables, changing only when that new data actually needs to be considered. That landmine effect to changes over time is one of the biggest arguments I make for never sending entities to a web client. (In addition to wasted time/payload sending data that clients don't need to see, and potential security issues if you accept entities back from the client.)