Retrieving entire data collection from a RavenDB

Question

I have a requirement where I need to fetch the entire data collection Users from RavenDB and compare the retrieved result set with another set of data. There are cl

Answer 1

This question was posted before this feature was available in RavenDB, but in case anyone else stumbles upon this now...

The encouraged way to do this is via the Streaming API. The RavenDB client batches the stream so it can automatically 'page' the requests/responses to/from the server. If you opt in to using the Streaming API, the client assumes you "know what you're doing" and does not check the 128/1024/30 limits that are used for regular queries.

var query = session.Query();
 
using (var enumerator = session.Advanced.Stream(query)) {
    while (enumerator.MoveNext()) {
        allUsers.Add(enumerator.Current.Document);
    }
}

var count = allUsers.Count;

Tip: Though this is the encouraged way to solve the problem... As a general rule it is best to avoid the situation to start with. What if there are a million records? That allUsers list is going to get huge. Maybe an index or transform could be done first to filter out what data you actually need to display to the user/process? Is this for reporting purposes? Maybe RavenDB should be automatically exporting to a SQL server with reporting services on it? Etc...