Creating a dynamic Linq select clause from Expressions

Question

Let's say I have defined the following variables:

IQueryable<MyClass> myQueryable;
Dictionary<string, Expression<Func<MyClass, bool>>> extraFields;
// the dictionary is keyed by a field name

Now, I want to tack on some dynamic fields to the IQueryable, so that it returns an IQueryable<ExtendedMyClass>, where ExtendedMyClass is defined as:

class ExtendedMyClass
{
  public MyClass MyObject {get; set;}
  public IEnumerable<StringAndBool> ExtraFieldValues {get; set;}
}

class StringAndBool
{
  public string FieldName {get; set;}
  public bool IsTrue {get; set;}
}

In other words, for every value in extraFields, I want to have a value in ExtendedMyClass.ExtraFieldValues representing whether that expression evaluates to True or not for that row.

I have a feeling this should be doable in dynamic Linq and LinqKit, though I've never used that seriously before. I'm also open to other suggestions, especially if this can be done in good ol' strong-typed Linq.

I am using Linq to Entities, so the query needs to translate to SQL.

Answer 1

So, we'll have a lot of steps here, but each individual step should be fairly short, self-contained, reusable, and relatively understandable.

The first thing we'll do is create a method that can combine expressions. What it will do is take an expression that accepts some input and generates an intermediate value. Then it will take a second expression that accepts, as input, the same input as the first, the type of the intermediate result, and then computes a new result. It will return a new expression taking the input of the first, and returning the output of the second.

public static Expression<Func<TFirstParam, TResult>>
    Combine<TFirstParam, TIntermediate, TResult>(
    this Expression<Func<TFirstParam, TIntermediate>> first,
    Expression<Func<TFirstParam, TIntermediate, TResult>> second)
{
    var param = Expression.Parameter(typeof(TFirstParam), "param");

    var newFirst = first.Body.Replace(first.Parameters[0], param);
    var newSecond = second.Body.Replace(second.Parameters[0], param)
        .Replace(second.Parameters[1], newFirst);

    return Expression.Lambda<Func<TFirstParam, TResult>>(newSecond, param);
}

To do this we simply replace all instances of the second parameter in the second expression's body with the body of the first expression. We also need to ensure both implementations use the same parameter instance for the main parameter.

This implementation requires having a method to replace all instances of one expression with another:

internal class ReplaceVisitor : ExpressionVisitor
{
    private readonly Expression from, to;
    public ReplaceVisitor(Expression from, Expression to)
    {
        this.from = from;
        this.to = to;
    }
    public override Expression Visit(Expression node)
    {
        return node == from ? to : base.Visit(node);
    }
}

public static Expression Replace(this Expression expression,
    Expression searchEx, Expression replaceEx)
{
    return new ReplaceVisitor(searchEx, replaceEx).Visit(expression);
}

Next we'll write a method that accepts a sequences of expressions that accept the same input and compute the same type of output. It will transform this into a single expression that accepts the same input, but computes a sequence of the output as a result, in which each item in the sequence represents the result of each of the input expressions.

This implementation is fairly straightforward; we create a new array, use the body of each expression (replacing the parameters with a consistent one) as each item in the array.

public static Expression<Func<T, IEnumerable<TResult>>> AsSequence<T, TResult>(
    this IEnumerable<Expression<Func<T, TResult>>> expressions)
{
    var param = Expression.Parameter(typeof(T));
    var body = Expression.NewArrayInit(typeof(TResult),
        expressions.Select(selector =>
            selector.Body.Replace(selector.Parameters[0], param)));
    return Expression.Lambda<Func<T, IEnumerable<TResult>>>(body, param);
}

Now that we have all of these general purpose helper methods out of the way, we can start working on your specific situation.

The first step here is to turn your dictionary into a sequence of expressions, each accepting a MyClass and creating a StringAndBool that represents that pair. To do this we'll use Combine on the value of the dictionary, and then use a lambda as the second expression to use it's intermediate result to compute a StringAndBool object, in addition to closing over the pair's key.

IEnumerable<Expression<Func<MyClass, StringAndBool>>> stringAndBools =
    extraFields.Select(pair => pair.Value.Combine((foo, isTrue) =>
        new StringAndBool()
        {
            FieldName = pair.Key,
            IsTrue = isTrue
        }));

Now we can use our AsSequence method to transform this from a sequence of selectors into a single selector that selects out a sequence:

Expression<Func<MyClass, IEnumerable<StringAndBool>>> extrafieldsSelector =
    stringAndBools.AsSequence();

Now we're almost done. We now just need to use Combine on this expression to write out our lambda for selecting a MyClass into an ExtendedMyClass while using the previous generated selector for selecting out the extra fields:

var finalQuery = myQueryable.Select(
    extrafieldsSelector.Combine((foo, extraFieldValues) =>
        new ExtendedMyClass
        {
            MyObject = foo,
            ExtraFieldValues = extraFieldValues,
        }));

---

We can take this same code, remove the intermediate variable and rely on type inference to pull it down to a single statement, assuming you don't find it too unweidly:

var finalQuery = myQueryable.Select(extraFields
    .Select(pair => pair.Value.Combine((foo, isTrue) =>
        new StringAndBool()
        {
            FieldName = pair.Key,
            IsTrue = isTrue
        }))
    .AsSequence()
    .Combine((foo, extraFieldValues) =>
        new ExtendedMyClass
        {
            MyObject = foo,
            ExtraFieldValues = extraFieldValues,
        }));

It's worth noting that a key advantage of this general approach is that the use of the higher level Expression methods results in code that is at least reasonably understandable, but also that can be statically verified, at compile time, to be type safe. There are a handful of general purpose, reusable, testable, verifiable, extension methods here that, once written, allows us to solve the problem purely through composition of methods and lambdas, and that doesn't require any actual expression manipulation, which is both complex, error prone, and removes all type safety. Each of these extension methods is designed in such a way that the resulting expression will always be valid, so long as the input expressions are valid, and the input expressions here are all known to be valid as they are lambda expressions, which the compiler verifies for type safety.

Answer 2

I think it's helpful here to take an example extraFields, imagine how would the expression that you need look like and then figure out how to actually create it.

So, if you have:

var extraFields = new Dictionary<string, Expression<Func<MyClass, bool>>>
{
    { "Foo", x => x.Foo },
    { "Bar", x => x.Bar }
};

Then you want to generate something like:

myQueryable.Select(
    x => new ExtendedMyClass
    {
        MyObject = x,
        ExtraFieldValues =
            new[]
            {
                new StringAndBool { FieldName = "Foo", IsTrue = x.Foo },
                new StringAndBool { FieldName = "Bar", IsTrue = x.Bar }
            }
    });

Now you can use the expression trees API and LINQKit to create this expression:

public static IQueryable<ExtendedMyClass> Extend(
    IQueryable<MyClass> myQueryable,
    Dictionary<string, Expression<Func<MyClass, bool>>> extraFields)
{
    Func<Expression<Func<MyClass, bool>>, MyClass, bool> invoke =
        LinqKit.Extensions.Invoke;

    var parameter = Expression.Parameter(typeof(MyClass));

    var extraFieldsExpression =
        Expression.Lambda<Func<MyClass, StringAndBool[]>>(
            Expression.NewArrayInit(
                typeof(StringAndBool),
                extraFields.Select(
                    field => Expression.MemberInit(
                        Expression.New(typeof(StringAndBool)),
                        new MemberBinding[]
                        {
                            Expression.Bind(
                                typeof(StringAndBool).GetProperty("FieldName"),
                                Expression.Constant(field.Key)),
                            Expression.Bind(
                                typeof(StringAndBool).GetProperty("IsTrue"),
                                Expression.Call(
                                    invoke.Method,
                                    Expression.Constant(field.Value),
                                    parameter))
                        }))),
            parameter);

    Expression<Func<MyClass, ExtendedMyClass>> selectExpression =
        x => new ExtendedMyClass
        {
            MyObject = x,
            ExtraFieldValues = extraFieldsExpression.Invoke(x)
        };

    return myQueryable.Select(selectExpression.Expand());
}