Generically Flatten Json using c#

Question

I want to generically flatten some json so I can convert to a datatable and bind to a datagrid using c#

What is the best way of doign it, bearing in mind I dont know

Answer 1

Looked also for a solution and that did the job for me. Used new C# 7 Tupels for the result set. If anyone have another lightweight solution i'm interested:-)

async Task Main()
{
    var jsonUsers = await new HttpClient().GetStringAsync(@"https://jsonplaceholder.typicode.com/users");

    foreach (var flattedChild in GetFlatJsonChilds(JToken.Parse(jsonUsers)))
        Console.WriteLine($"{flattedChild.path}: {flattedChild.value}");
}

IEnumerable<(string path, string value)> GetFlatJsonChilds(JToken token)
{
    foreach (var child in token.Children())
    {   
        if(token.Type != JTokenType.Array &&token.Children().First().Type != JTokenType.Property && !child.Children().Any())
            yield return (child.Path, child.ToString());
        foreach(var childChild in GetFlatJsonChilds(child))
            yield return childChild;
    }
}

The Result for https://jsonplaceholder.typicode.com/users :

[0].id: 1
[0].name: Leanne Graham
[0].username: Bret
[0].email: Sincere@april.biz
[0].address.street: Kulas Light
[0].address.suite: Apt. 556
[0].address.city: Gwenborough
[0].address.zipcode: 92998-3874
[0].address.geo.lat: -37.3159
[0].address.geo.lng: 81.1496
[0].phone: 1-770-736-8031 x56442
[0].website: hildegard.org
[0].company.name: Romaguera-Crona
[0].company.catchPhrase: Multi-layered client-server neural-net
[0].company.bs: harness real-time e-markets
[1].id: 2
[1].name: Ervin Howell
[1].username: Antonette
[1].email: Shanna@melissa.tv
[1].address.street: Victor Plains
[1].address.suite: Suite 879
[1].address.city: Wisokyburgh
[1].address.zipcode: 90566-7771
[1].address.geo.lat: -43.9509
[1].address.geo.lng: -34.4618
[1].phone: 010-692-6593 x09125
[1].website: anastasia.net
[1].company.name: Deckow-Crist
[1].company.catchPhrase: Proactive didactic contingency
[1].company.bs: synergize scalable supply-chains
[2].id: 3
[2].name: Clementine Bauch
[2].username: Samantha
[2].email: Nathan@yesenia.net
[2].address.street: Douglas Extension
[2].address.suite: Suite 847
[2].address.city: McKenziehaven
[2].address.zipcode: 59590-4157
[2].address.geo.lat: -68.6102
[2].address.geo.lng: -47.0653
[2].phone: 1-463-123-4447
[2].website: ramiro.info
[2].company.name: Romaguera-Jacobson
[2].company.catchPhrase: Face to face bifurcated interface
[2].company.bs: e-enable strategic applications
[3].id: 4
[3].name: Patricia Lebsack
[3].username: Karianne
[3].email: Julianne.OConner@kory.org
[3].address.street: Hoeger Mall
[3].address.suite: Apt. 692
[3].address.city: South Elvis
[3].address.zipcode: 53919-4257
[3].address.geo.lat: 29.4572
[3].address.geo.lng: -164.2990
[3].phone: 493-170-9623 x156
[3].website: kale.biz
[3].company.name: Robel-Corkery
[3].company.catchPhrase: Multi-tiered zero tolerance productivity
[3].company.bs: transition cutting-edge web services
[4].id: 5
[4].name: Chelsey Dietrich
[4].username: Kamren
[4].email: Lucio_Hettinger@annie.ca
[4].address.street: Skiles Walks
[4].address.suite: Suite 351
[4].address.city: Roscoeview
[4].address.zipcode: 33263
[4].address.geo.lat: -31.8129
[4].address.geo.lng: 62.5342
[4].phone: (254)954-1289
[4].website: demarco.info
[4].company.name: Keebler LLC
[4].company.catchPhrase: User-centric fault-tolerant solution
[4].company.bs: revolutionize end-to-end systems
[5].id: 6
[5].name: Mrs. Dennis Schulist
[5].username: Leopoldo_Corkery
[5].email: Karley_Dach@jasper.info
[5].address.street: Norberto Crossing
[5].address.suite: Apt. 950
[5].address.city: South Christy
[5].address.zipcode: 23505-1337
[5].address.geo.lat: -71.4197
[5].address.geo.lng: 71.7478
[5].phone: 1-477-935-8478 x6430
[5].website: ola.org
[5].company.name: Considine-Lockman
[5].company.catchPhrase: Synchronised bottom-line interface
[5].company.bs: e-enable innovative applications
[6].id: 7
[6].name: Kurtis Weissnat
[6].username: Elwyn.Skiles
[6].email: Telly.Hoeger@billy.biz
[6].address.street: Rex Trail
[6].address.suite: Suite 280
[6].address.city: Howemouth
[6].address.zipcode: 58804-1099
[6].address.geo.lat: 24.8918
[6].address.geo.lng: 21.8984
[6].phone: 210.067.6132
[6].website: elvis.io
[6].company.name: Johns Group
[6].company.catchPhrase: Configurable multimedia task-force
[6].company.bs: generate enterprise e-tailers
[7].id: 8
[7].name: Nicholas Runolfsdottir V
[7].username: Maxime_Nienow
[7].email: Sherwood@rosamond.me
[7].address.street: Ellsworth Summit
[7].address.suite: Suite 729
[7].address.city: Aliyaview
[7].address.zipcode: 45169
[7].address.geo.lat: -14.3990
[7].address.geo.lng: -120.7677
[7].phone: 586.493.6943 x140
[7].website: jacynthe.com
[7].company.name: Abernathy Group
[7].company.catchPhrase: Implemented secondary concept
[7].company.bs: e-enable extensible e-tailers
[8].id: 9
[8].name: Glenna Reichert
[8].username: Delphine
[8].email: Chaim_McDermott@dana.io
[8].address.street: Dayna Park
[8].address.suite: Suite 449
[8].address.city: Bartholomebury
[8].address.zipcode: 76495-3109
[8].address.geo.lat: 24.6463
[8].address.geo.lng: -168.8889
[8].phone: (775)976-6794 x41206
[8].website: conrad.com
[8].company.name: Yost and Sons
[8].company.catchPhrase: Switchable contextually-based project
[8].company.bs: aggregate real-time technologies
[9].id: 10
[9].name: Clementina DuBuque
[9].username: Moriah.Stanton
[9].email: Rey.Padberg@karina.biz
[9].address.street: Kattie Turnpike
[9].address.suite: Suite 198
[9].address.city: Lebsackbury
[9].address.zipcode: 31428-2261
[9].address.geo.lat: -38.2386
[9].address.geo.lng: 57.2232
[9].phone: 024-648-3804
[9].website: ambrose.net
[9].company.name: Hoeger LLC
[9].company.catchPhrase: Centralized empowering task-force
[9].company.bs: target end-to-end models

Answer 2

Another variant using Newtonsoft's Json.NET LINQ to JSON for object at root (the same can be done with JArray also):

var flattened = JObject.Parse(json)
    .Descendants()
    .OfType<JValue>()
    .ToDictionary(jv => jv.Path, jv => jv.ToString())

Answer 3

Using the library Json.Net You could use the JSONPath $..* to get all members of the JSON structure and filter out the ones with no children to skip the container properties.

e.g.

var schemaObject = JObject.Parse(schema);
var values = schemaObject
    .SelectTokens("$..*")
    .Where(t => !t.HasValues)
    .ToDictionary(t => t.Path, t => t.ToString());

Answer 4

Deserialize, then LINQ select to flatten. I'm guessing, since you haven't stated it, that you want all the appointment and installer information on the same record as the specific installation?

My initial idea would be to leverage dynamics so that you can avoid having to put in static schemas for your JSON. If you already have static types that can act as JSON schemas, then you can avoid dynamics (and all that it entails). Here's an example class - using JSON.NET - to illustrate what I'm thinking:

public class DeserializeAndFlatten
{
    public dynamic ParseJson()
    {
        var appointment = JObject.Parse(JsonData.JSON_TO_PARSE);  // <-- replace the constant w/ the real JSON...

        // this is where you flatten it all out!
        // not going to put all the fields in, that would kill the example, LOL
        var installations = appointment["installations"].Select(installation => new
        {
            appointmentId = appointment["appointmentid"],
            policyId = appointment["policyid"],

            vehicleId = appointment["vehicle"]["id"],
            vehicleMake = appointment["vehicle"]["make"],
            vehicleModel = appointment["vehicle"]["model"],

            installerId = appointment["installer"]["installerid"],
            installerName = appointment["installer"]["name"],

            installationId = installation["installationid"],
            installationStatus = installation["installationstatus"]["installationstatus"],
            installationStatusId = installation["installationstatus"]["installationstatusid"],
        }).ToList();

        return installations;
    }
}

You can test the code:

    static void Main(string[] args)
    {
        var jsonParser = new DeserializeAndFlatten();
        var installations = jsonParser.ParseJson();

        // FYI we get back a dynamic listing, 
        // so intellisense wont work...
        foreach (var installation in installations)
        {
            Console.WriteLine($"appointmentId: {installation.appointmentId}");
            Console.WriteLine($"installer: {installation.installerName}");
            Console.WriteLine($"installation id: {installation.installationId}");
            Console.WriteLine($"status: {installation.installationStatus}");
            Console.WriteLine();
        }

        Console.ReadLine();
    }

Answer 5

Here is one another way to flatten JSON / convert to DataTable using Cinchoo ETL

Flatten JSON:

using (var r = new ChoJSONReader("*** JSON file path ***"))
{
    foreach (var rec in r.Select(f => f.Flatten()))
        Console.WriteLine(rec.Dump());
}

JSON to DataTable:

using (var r = new ChoJSONReader("*** JSON file path ***"))
{
    var dt = r.AsDataTable();
    Console.WriteLine(dt.DumpAsJson());
}

Answer 6

For those who need same in F#:

module JsonFlatten =

let Join prefix name =
    if String.IsNullOrEmpty(prefix) then name else prefix + "." + name

let rec FillDictionaryFromJToken (dict:Dictionary<string, string>) (token:JToken) (prefix:string) =
    match token.Type with
    | JTokenType.Object ->
        for prop in token.Children<JProperty>() do
            FillDictionaryFromJToken dict prop.Value (Join prefix prop.Name)
    | JTokenType.Array ->
        let mutable index = 0

        for value in token.Children() do
            FillDictionaryFromJToken dict  value (Join prefix (index.ToString()))
            index <- index + 1
    | _ ->
        dict.Add(prefix, sprintf "%A" (token :?> JValue).Value)

let DeserializeAndFlatten(json:string) =
    let dict = Dictionary<string, string>()
    let token = JToken.Parse(json);
    FillDictionaryFromJToken dict  token ""
    dict