Customizing Json.NET serialization: turning object into array to avoid repetition of property names

Question

I\'m sending large amounts of different JSON graphs from a server to a client (I control both) and they all contain a pathological case: a large array of homogeneous (same t

Answer 1

You can achieve what you want by using Custom JsonConverter. Lets say you have the following test class:

public class MyTestClass
{
    public MyTestClass(int key1, string key2, decimal key3)
    {
        m_key1 = key1;
        m_key2 = key2;
        m_key3 = key3;
    }

    private int m_key1;
    public int Key1 { get { return m_key1; } }

    private string m_key2;
    public string Key2 { get { return m_key2; } }

    private decimal m_key3;
    public decimal Key3 { get { return m_key3; } }
}

This solution assumes that you'll work with List<MyTestClass> all the time, but it is not tied to the type MyTestClass. It is a generic solution that can work with any List<T>, but the type T has get only properties and has a constructor that sets all property values.

var list = new List<MyTestClass>
            {
                new MyTestClass
                {
                    Key1 = 1,
                    Key2 = "Str 1",
                    Key3 = 8.3m
                },
                new MyTestClass
                {
                    Key1 = 72,
                    Key2 = "Str 2",
                    Key3 = 134.8m
                },
                new MyTestClass
                {
                    Key1 = 99,
                    Key2 = "Str 3",
                    Key3 = 91.45m
                }
            };

If you serialize this list with the usual JSON.NET serialization the result would be:

[{"Key1":1,"Key2":"Str 1","Key3":8.3},{"Key1":72,"Key2":"Str 2","Key3":134.8},{"Key1":99,"Key2":"Str 3","Key3":91.45}]

That's not what you expect. From what you posted, the desired result for you is:

[["Key1","Key2","Key3"],[1,"Str 1",8.3],[72,"Str 2",134.8],[99,"Str 3",91.45]]

where the first inner array represents the key names and starting from the second to the last are the values of each property of each object from the list. You can achieve this kind of serialization by writing custom JsonConverter:

public class CustomJsonConverter : JsonConverter
{

    public override bool CanConvert(Type objectType)
    {
        return true;
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        if (!(objectType.IsGenericType)) return null;

        var deserializedList = (IList)Activator.CreateInstance(objectType);
        var jArray = JArray.Load(reader);

        var underlyingType = objectType.GetGenericArguments().Single();
        var properties = underlyingType.GetProperties();

        Type[] types = new Type[properties.Length];

        for (var i = 0; i < properties.Length; i++)
        {
            types[i] = properties[i].PropertyType;
        }

        var values = jArray.Skip(1);

        foreach (JArray value in values)
        {
            var propertiesValues = new object[properties.Length];

            for (var i = 0; i < properties.Length; i++)
            {
                propertiesValues[i] = Convert.ChangeType(value[i], properties[i].PropertyType);
            }

            var constructor = underlyingType.GetConstructor(types);
            var obj = constructor.Invoke(propertiesValues);
            deserializedList.Add(obj);
        }

        return deserializedList;
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        if (!(value.GetType().IsGenericType) || !(value is IList)) return;
        var val = value as IList;

        PropertyInfo[] properties = val.GetType().GetGenericArguments().Single().GetProperties();

        writer.WriteStartArray();


        writer.WriteStartArray();

        foreach (var p in properties)
            writer.WriteValue(p.Name);

        writer.WriteEndArray();

        foreach (var v in val)
        {
            writer.WriteStartArray();

            foreach (var p in properties)
                writer.WriteValue(v.GetType().GetProperty(p.Name).GetValue(v));

            writer.WriteEndArray();
        }

        writer.WriteEndArray();
    }
}

and use the following line for serialization:

var jsonStr = JsonConvert.SerializeObject(list, new CustomJsonConverter());

To deserialize the string into a list of objects from typeof(MyTestClass), use the following line:

var reconstructedList = JsonConvert.DeserializeObject<List<MyTestClass>>(jsonStr, new CustomJsonConverter());

You can use the CustomJsonConverter with any generic list of objects. Please note that this solution assumes that the order of the properties during serialization and deserialization is the same.

Answer 2

To answer your first question: yes, someone already built this and called it 'jsonh'.

Bad thing about it is: it's not available for c# but you have enough code there to implement it yourself... I have not yet seen it as a ready made package for C# anywhere

and then there is another "standard" that almost does this, but is meant for exactly the same: rjson

And again: no C#...

If you just (g)zip your json data, it will automatically achieve the kind of compression you want (but better) since, as you already stated huffman, it uses a huffman tree. And the idea behind jsonh and rjson is to avoid duplication in the keys, while gzip would make a difference between keys, values or other glyphs.

Answer 3

I believe the best way to achieve what you are looking for is to use a custom JsonConverter as was suggested by @Ilija Dimov. His converter is a good start, and should work fine for certain cases, but you may run into trouble if you are serializing a more complex graph of objects. I offer the following converter as an alternative solution. This converter has the following advantages:

• Uses the Json.Net's built-in serialization logic for the list items, so that any attributes applied to the classes are respected, including [JsonConstructor] and [JsonProperty]. Other converters are respected as well.
• Ignores lists of primitives and strings so that these are serialized normally.
• Supports List<YourClass> where YourClass contains complex objects, including List<YourOtherClass>.

Limitations:

• Does not currently support lists of anything enumerable, e.g. List<List<YourClass>> or List<Dictionary<K, YourClass>>, but could be modified to do so if needed. These will be serialized in the usual way for now.

Here is the code for the converter:

class ListCompactionConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        // We only want to convert lists of non-enumerable class types (including string)
        if (objectType.IsGenericType && objectType.GetGenericTypeDefinition() == typeof(List<>))
        {
            Type itemType = objectType.GetGenericArguments().Single();
            if (itemType.IsClass && !typeof(IEnumerable).IsAssignableFrom(itemType))
            {
                return true;
            }
        }
        return false;
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        JArray array = new JArray();
        IList list = (IList)value;
        if (list.Count > 0)
        {
            JArray keys = new JArray();

            JObject first = JObject.FromObject(list[0], serializer);
            foreach (JProperty prop in first.Properties())
            {
                keys.Add(new JValue(prop.Name));
            }
            array.Add(keys);

            foreach (object item in list)
            {
                JObject obj = JObject.FromObject(item, serializer);
                JArray itemValues = new JArray();
                foreach (JProperty prop in obj.Properties())
                {
                    itemValues.Add(prop.Value);
                }
                array.Add(itemValues);
            }
        }
        array.WriteTo(writer);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        IList list = (IList)Activator.CreateInstance(objectType);  // List<T>
        JArray array = JArray.Load(reader);
        if (array.Count > 0)
        {
            Type itemType = objectType.GetGenericArguments().Single();

            JArray keys = (JArray)array[0];
            foreach (JArray itemValues in array.Children<JArray>().Skip(1))
            {
                JObject item = new JObject();
                for (int i = 0; i < keys.Count; i++)
                {
                    item.Add(new JProperty(keys[i].ToString(), itemValues[i]));
                }

                list.Add(item.ToObject(itemType, serializer));
            }
        }
        return list;
    }
}

Below is a full round-trip demo using this converter. We have a list of mutable Company objects which each contain a list of immutable Employees. For demonstration purposes, each company also has a simple list of string aliases using a custom JSON property name, and we also use an IsoDateTimeConverter to customize the date format for the employee HireDate. The converters are passed to the serializer via the JsonSerializerSettings class.

class Program
{
    static void Main(string[] args)
    {
        List<Company> companies = new List<Company>
        {
            new Company
            {
                Name = "Initrode Global",
                Aliases = new List<string> { "Initech" },
                Employees = new List<Employee>
                {
                    new Employee(22, "Bill Lumbergh", new DateTime(2005, 3, 25)),
                    new Employee(87, "Peter Gibbons", new DateTime(2011, 6, 3)),
                    new Employee(91, "Michael Bolton", new DateTime(2012, 10, 18)),
                }
            },
            new Company
            {
                Name = "Contoso Corporation",
                Aliases = new List<string> { "Contoso Bank", "Contoso Pharmaceuticals" },
                Employees = new List<Employee>
                {
                    new Employee(23, "John Doe", new DateTime(2007, 8, 22)),
                    new Employee(61, "Joe Schmoe", new DateTime(2009, 9, 12)),
                }
            }
        };

        JsonSerializerSettings settings = new JsonSerializerSettings();
        settings.Converters.Add(new ListCompactionConverter());
        settings.Converters.Add(new IsoDateTimeConverter { DateTimeFormat = "dd-MMM-yyyy" });
        settings.Formatting = Formatting.Indented;

        string json = JsonConvert.SerializeObject(companies, settings);
        Console.WriteLine(json);
        Console.WriteLine();

        List<Company> list = JsonConvert.DeserializeObject<List<Company>>(json, settings);
        foreach (Company c in list)
        {
            Console.WriteLine("Company: " + c.Name);
            Console.WriteLine("Aliases: " + string.Join(", ", c.Aliases));
            Console.WriteLine("Employees: ");
            foreach (Employee emp in c.Employees)
            {
                Console.WriteLine("  Id: " + emp.Id);
                Console.WriteLine("  Name: " + emp.Name);
                Console.WriteLine("  HireDate: " + emp.HireDate.ToShortDateString());
                Console.WriteLine();
            }
            Console.WriteLine();
        }
    }
}

class Company
{
    public string Name { get; set; }
    [JsonProperty("Doing Business As")]
    public List<string> Aliases { get; set; }
    public List<Employee> Employees { get; set; }
}

class Employee
{
    [JsonConstructor]
    public Employee(int id, string name, DateTime hireDate)
    {
        Id = id;
        Name = name;
        HireDate = hireDate;
    }
    public int Id { get; private set; }
    public string Name { get; private set; }
    public DateTime HireDate { get; private set; }
}

Here is the output from the above demo, showing the intermediate JSON as well as the contents of the objects deserialized from it.

[
  [
    "Name",
    "Doing Business As",
    "Employees"
  ],
  [
    "Initrode Global",
    [
      "Initech"
    ],
    [
      [
        "Id",
        "Name",
        "HireDate"
      ],
      [
        22,
        "Bill Lumbergh",
        "25-Mar-2005"
      ],
      [
        87,
        "Peter Gibbons",
        "03-Jun-2011"
      ],
      [
        91,
        "Michael Bolton",
        "18-Oct-2012"
      ]
    ]
  ],
  [
    "Contoso Corporation",
    [
      "Contoso Bank",
      "Contoso Pharmaceuticals"
    ],
    [
      [
        "Id",
        "Name",
        "HireDate"
      ],
      [
        23,
        "John Doe",
        "22-Aug-2007"
      ],
      [
        61,
        "Joe Schmoe",
        "12-Sep-2009"
      ]
    ]
  ]
]

Company: Initrode Global
Aliases: Initech
Employees:
  Id: 22
  Name: Bill Lumbergh
  HireDate: 3/25/2005

  Id: 87
  Name: Peter Gibbons
  HireDate: 6/3/2011

  Id: 91
  Name: Michael Bolton
  HireDate: 10/18/2012


Company: Contoso Corporation
Aliases: Contoso Bank, Contoso Pharmaceuticals
Employees:
  Id: 23
  Name: John Doe
  HireDate: 8/22/2007

  Id: 61
  Name: Joe Schmoe
  HireDate: 9/12/2009

I've added a fiddle here in case you'd like to play with the code.

Answer 4

Manatee.Json can do direct JSON-JSON transforms without the mess of special serialization converters. It's a target-first approach that uses JSONPath to identify the specific elements within the source data.

For reference, your source data:

[{"Key1":87,"Key2":99},{"Key1":42,"Key2":-8}]

We then define a template:

[
  ["Key1","Key2"],
  ["$[*]","$.Key1"],
  ["$[*]","$.Key2"]
]

This will map your source data to:

[["Key1","Key2"],[87,42],[99,-8]]

just as you wanted.

The template is based on jsonpath-object-transform. Here's how it works:

• For the most part, the template has the same shape as your target.
• For each property, you specify a JSON Path that identifies the data in the source. (Object property mapping isn't directly shown in this example since you only have arrays, but the link above has a few.)
• There is a special case for arrays. If the array has two elements, and the first element is a JSON Path, then the second array is interpreted as the template for each item in the array. Otherwise, the array is copied as is, mapping data from the source as normal when the element is a path.

So for your case (forgive the C-style comments in the JSON),

[                     // Root is an array.
  ["Key1","Key2"],    // Array literal.
  ["$[*]","$.Key1"],  // Take all of the elements in the original array '$[*]'
                      //   and use the value under the "Key1" property '$.Key1'
  ["$[*]","$.Key2"]   // Similiarly for the "Key2" property
]

Note There is an edge case where you want to map a value to a literal array with two elements. This will not work properly.

Once mapped, you can deserialize however you like (Manatee.Json can do that for you, too.).

Edit

I realized that I didn't put any code in my answer, so here it is.

JsonValue source = new JsonArray
    {
        new JsonObject {{"Key1", 87}, {"Key2", 99}},
        new JsonObject {{"Key1", 42}, {"Key2", -8}}
    };
JsonValue template = new JsonArray
    {
        new JsonArray {"Key1", "Key2"},
        new JsonArray {"$[*]", "$.Key1"},
        new JsonArray {"$[*]", "$.Key2"}
    };
var result = source.Transform(template);

That's it.

Edit 2

I was having trouble devising a reverse translation, so here's how you would do this with serialization only.

You would need to register a couple methods to perform the mapping and serialization yourself. Essentially, you instruct the serializer how to build and descontruct the JSON.

Your data model:

public class MyData
{
    public int Key1 { get; set; }
    public int Key2 { get; set; }
}

The serialization methods:

public static class MyDataListSerializer
{
    public static JsonValue ToJson(List<MyData> data, JsonSerializer serializer)
    {
        return new JsonArray
            {
                new JsonArray {"Key1", "Key2"},
                new JsonArray(data.Select(d => d.Key1)),
                new JsonArray(data.Select(d => d.Key2)),
            };
    }
    public static MyData FromJson(JsonValue value, JsonSerializer serializer)
    {
        return value.Array.Skip(1)
                    .Array.Select((jv, i) => new MyData
                                             {
                                                 Key1 = (int) jv.Number,
                                                 Key2 = value.Array[2].Array[i]
                                             };
    }
}

Registering the methods:

JsonSerializationTypeRegistry.RegisterType(MyDataSerializer.ToJson,
                                           MyDataSerializer.FromJson);

And finally the deserialize methods. I'm not sure what your method signatures are, but you mentioned that you're receiving a stream for the deserialize, so I'll start with that.

public string Serialize(MyData data)
{
    // _serializer is an instance field of type JsonSerializer
    return _serializer.Serialize(data).ToString();
}
public MyData Deserialize(Stream stream)
{
    var json = JsonValue.Parse(stream);
    return _serializer.Deserialize<MyData>(json);
}

This approach forces the static serializer methods to handle the formatting of the JSON. There is no real transformation occurring here; it's serializing directly to and from the desired format.

Edit 3

Hopefully this is the last edit. This answer is becoming a dissertation.

I couldn't live with myself not having a translation solution. However, working out the serialization piece led me to the answer. There was ambiguity in how the transformer interpreted the paths in that array special case, so I split it up.

JsonPath specifies an alternate root symbol when looking at items within an array: @. This convention is now adopted in the transformer as well.

The original transform template becomes:

[["Key1","Key2"],["$[*]","@.Key1"],["$[*]","@.Key2"]]

This allows us to create a reverse template:

[
    "$[1][*]",             // Get all of the items in the first value list
    {
        "Key1":"@",        // Key1 is sourced from the item returned by '$[1][*]'
        "Key2":"$[2][*]"   // Key2 is sourced from the items in the second element
                           // of the original source (not the item returned by '$[1][*]')
    }
]

Now you can transform both directions and you don't have to do anything fancy with the custom serialize methods.

The serializer will look something like this now:

public string Serialize(MyData data)
{
    // _serializer is an instance field of type JsonSerializer
    var json = _serializer.Serialize(data);
    // _transformTemplate is an instance field of type JsonValue
    // representing the first template from above.
    var transformedJson = json.Transform(_transformTemplate);
    return transformedJson.ToString();
}
public MyData Deserialize(Stream stream)
{
    var json = JsonValue.Parse(stream);
    // _reverseTransformTemplate is an instance field of type JsonValue
    // representing the second template from above.
    var untransformedJson = json.Transform(_reverseTransformTemplate);
    return _serializer.Deserialize<MyData>(untransformedJson);
}

Answer 5

No need for custom JSON converters. Just make your class implement IEnumerable<object>. Json.NET will then serialize your data as an array instead of an object.

For instance, instead of...

// will be serialized as: {"Key1":87,"Key2":99}
public class Foo
{
    public string Key1;
    public string Key2;
}

...write this:

// will be serialized as: [87,99]
public class Foo : IEnumerable<object>
{
    public string Key1;
    public string Key2;

    IEnumerator<object> IEnumerable<object>.GetEnumerator() => EnumerateFields().GetEnumerator();
    IEnumerator IEnumerable.GetEnumerator() => EnumerateFields().GetEnumerator();
    IEnumerable<object> EnumerateFields()
    {
        yield return Key1;
        yield return Key2;
    }
}

If you need to apply this strategy to many classes then you can declare an abstract base class to get rid of some of the boilerplate:

// Base class for objects to be serialized as "[...]" instead of "{...}"
public abstract class SerializedAsArray : IEnumerable<object>
{
    IEnumerator<object> IEnumerable<object>.GetEnumerator() =>
        EnumerateFields().GetEnumerator();
    IEnumerator IEnumerable.GetEnumerator() =>
        EnumerateFields().GetEnumerator();
    protected abstract IEnumerable<object> EnumerateFields();
}

// will be serialized as: [87,99]
public class Foo : SerializedAsArray
{
    public string Key1;
    public string Key2;

    protected override IEnumerable<object> EnumerateFields()
    {
        yield return Key1;
        yield return Key2;
    }
}

Answer 6

A big strength of the popular JSON serialization libraries (not to say the whole idea behind JSON) is to take language features - objects, arrays, literals - and serialize them into an equivalent JSON representation. You can look at an object structure in C# (e.g.) and you know what the JSON will look like. This is not the case if you start changing the whole serialization mechanism. *)

Apart from DoXicK's suggestion to use gzip for compression, if you really want to define a different JSON format, why not simply transform your object tree in C# before serializing it?

Something like

var input = new[]
    {
        new { Key1 = 87, Key2 = 99 },
        new { Key1 = 42, Key2 = -8 }
    };


var json = JSON.Serialize(Transform(input));


object Transform(object[] input)
{
    var props = input.GetProperties().ToArray();
    var keys = new[] { "$" }.Concat(props.Select(p => p.Name));
    var stripped = input.Select(o => props.Select(p => p.GetValue(o)).ToArray();
    return keys.Concat(stripped);
}

would do. This way you will not confuse any programmers by changing the way JSON works. Instead the transformation will be an explicit pre-/postprocessing step.

---

*) I would even argue that it is like a protocol: Object is { }, array is [ ]. It is, as the name says, a serialization of your object structure. If you change the serialization mechanism, you change the protocol. Once you do that, you don't need to look it like JSON any longer at all, because the JSON does not properly represent your object structure anyway. Calling it JSON and making it look like such has the potential to confuse each of your fellow/future programmers as well as yourself when you re-visit your code later.