Entity Framework 6: is there a way to iterate through a table without holding each row in memory

Question

I would like to be able to iterate through every row in an entity table without holding every row in memory. This is a read only operation and every row can be discarded aft

Answer 1

I have done this by pages. And cleaning the Context after each page load.

Sample: Load first 50 rows Iterate over them Clean the Context or create a new one.

Load second 50 rows ...

Clean the Context = Set all its Entries as Detached.

Answer 2

I haven't tested it, but try foreach (Quote L in context.Quotes.AsNoTracking()) {...}. .AsNoTracking() should not put entities in cache so I assume they will be consumed by GC when they out of the scope.

Another option is to use context.Entry(quote).State = EntityState.Detached; in the foreach loop. Should have the similar effect as the option 1.

Third option (should definitely work, but require more coding) would be to implement batch processing (select top N entities, process, select next top N). In this case make sure that you dispose and create new context every iteration (so GC can eat it:)) and use proper OrderBy() in the query.

Answer 3

Based on your last comment, I'm still confused. Take a look at both of below code.

EF

using (var ctx = new AppContext())
{
    foreach (var order in ctx.Orders)
    {
        Console.WriteLine(order.Date);
    }
}

Data Reader

var constr = ConfigurationManager.ConnectionStrings["AppContext"].ConnectionString;
using (var con = new SqlConnection(constr))
{
    con.Open();    
    var cmd = new SqlCommand("select * from dbo.Orders", con);
    var reader = cmd.ExecuteReader();
    while (reader.Read())
    {
        Console.WriteLine(reader["Date"]);
    }
}

Even though EF has few initial query, both of them execute similar query that can be seen from profiler..

Answer 4

You need to use an EntityDataReader, which behaves in a way similar to a traditional ADO.NET DataReader.

The problem is that, to do so, you need to use ObjectContext instead of DbContext, which makes things harder.

See this SO answer, not the acepted one: How can I return a datareader when using Entity Framework 4?

Even if this referes to EF4, in EF6 things work in the same way. Usually an ORM is not intended for streaming data, that's why this functionality is so hidden.

You can also look at this project: Entity Framework (Linq to Entities) to IDataReader Adapter