EntityFramework : Calling ToList() on IQueryable with ~11.000 records takes 10 seconds

Question

I want to return a relatively large number of records from SQL Express 2008 R2 server, via EntityFramework 4 through WCF service to a WCF client. My test table contains around 11.000 records at the moment. The LINQ query is as simple as this:

Database DB = new Database(); // create object context
var retValue = DB.Entities.Persons
        .Include("District")
        .Include("District.City")
        .Include("District.City.State")
        .Include("Nationality")

return retValue.ToList();

This takes about 10 seconds to complete.

The same SELECT query takes less than 1 second when executed in SQL Server Managament Studio.

Does it have to be that slow in EF ?

Answer 1

Your query is not simple, it contains a lot of joins (due to the Includes) and more importantly it might return a lot of duplicated data, especially if the included navigation properties are collections: https://stackoverflow.com/a/5522195/270591

The time comsuming part is object materialization and attaching the entities to the context when the result from the database is returned to the Entity Framework context.

This is confirmed by your measurements (in the comments to your question) that a second query within the same context is very fast. In this case EF will perform a query to the database but doesn't need to materialize the objects again because they are still attached to the context.

If you run the second query in a second context the resulting entities must the attached to the new context - and this step is again slow (also confirmed by your measurements).

This is probably a point where a query with EF is in fact slow and adds a lot of overhead compared to a raw SQL query. EF needs to create many data structures prepared for change tracking and managing object identities in the context which consumes additional time.

The only way I can see to improve the performance is disabling change tracking (supposed, you don't need it for your operations). In EF 4.0 / ObjectContext it would be:

Database DB = new Database();
DB.Entities.Persons.MergeOption = MergeOption.NoTracking;
// MergeOption is in System.Data.Objects namespace

When using this approach, one has to be aware though that related objects will be created as separate objects even when they have the same key - which is not the case with enabled change tracking because attaching to the context will avoid this duplication.

So, potentially more objects will be loaded into memory. If this is counterproductive and degrades actually the performance even more or if it still performs better is a matter of a test.

Answer 2

This is very likely because query compilation (LINQ-query w lots of includes -> SQL to use) is very slow in EF compared to query execution. You can verify if this is the problem by CPU profiling your code. Consider using fewer includes + multiple smaller queries, using compiled query/ies or upgrade to the latest EF5 beta.