How to get records in EF that match a list of combinations (key/values)?

Question

I have a database table with records for each user/year combination.

How can I get data from the database using EF and a list of userId/year combinations? S

Answer 1

This is a notorious problem that I discussed before here. Krishna Muppalla's solution is among the solutions I came up with there. Its disadvantage is that it's not sargable, i.e. it can't benefit from any indexes on the involved database fields.

In the meantime I coined another solution that may be helpful in some circumstances. Basically it groups the input data by one of the fields and then finds and unions database data by grouping key and a Contains query of group elements:

IQueryable items = null;

foreach (var yearUserIds in userYears.GroupBy(t => t.Year, t => t.UserId))
{
    var userIds = yearUserIds.ToList();
    var grp = dbcontext.YearResults
        .Where(x => x.Year == yearUserIds.Key 
                 && userIds.Contains(x.UserId));
    items = items == null ? grp : items.Concat(grp);
}

I use Concat here because Union will waste time making results distinct and in EF6 Concat will generate SQL with chained UNION statements while Union generates nested UNION statements and the maximum nesting level may be hit.

This query may perform well enough when indexes are in place. In theory, the maximum number of UNIONs in a SQL statement is unlimited, but the number of items in an IN clause (that Contains translates to) should not exceed a couple of thousands. That means that the content of your data will determine which grouping field performs better, Year or UserId. The challenge is to minimize the number of UNIONs while keeping the number of items in all IN clauses below approx. 5000.