Flatten nested dictionaries, compressing keys

Question

Suppose you have a dictionary like:

{\'a\': 1,
 \'c\': {\'a\': 2,
       \'b\': {\'x\': 5,
             \'y\' : 10}},
 \'d\': [1, 2, 3]}

Ho

Answer 1

The answers above work really well. Just thought I'd add the unflatten function that I wrote:

def unflatten(d):
    ud = {}
    for k, v in d.items():
        context = ud
        for sub_key in k.split('_')[:-1]:
            if sub_key not in context:
                context[sub_key] = {}
            context = context[sub_key]
        context[k.split('_')[-1]] = v
    return ud

Note: This doesn't account for '_' already present in keys, much like the flatten counterparts.

Answer 2

here's a solution using a stack. No recursion.

def flatten_nested_dict(nested):
    stack = list(nested.items())
    ans = {}
    while stack:
        key, val = stack.pop()
        if isinstance(val, dict):
            for sub_key, sub_val in val.items():
                stack.append((f"{key}_{sub_key}", sub_val))
        else:
            ans[key] = val
    return ans

Answer 3

Basically the same way you would flatten a nested list, you just have to do the extra work for iterating the dict by key/value, creating new keys for your new dictionary and creating the dictionary at final step.

import collections

def flatten(d, parent_key='', sep='_'):
    items = []
    for k, v in d.items():
        new_key = parent_key + sep + k if parent_key else k
        if isinstance(v, collections.MutableMapping):
            items.extend(flatten(v, new_key, sep=sep).items())
        else:
            items.append((new_key, v))
    return dict(items)

>>> flatten({'a': 1, 'c': {'a': 2, 'b': {'x': 5, 'y' : 10}}, 'd': [1, 2, 3]})
{'a': 1, 'c_a': 2, 'c_b_x': 5, 'd': [1, 2, 3], 'c_b_y': 10}

Answer 4

If you're using pandas there is a function hidden in pandas.io.json._normalize¹ called nested_to_record which does this exactly.

from pandas.io.json._normalize import nested_to_record    

flat = nested_to_record(my_dict, sep='_')

---

¹ In pandas versions 0.24.x and older use pandas.io.json.normalize (without the _)

Answer 5

Code:

test = {'a': 1, 'c': {'a': 2, 'b': {'x': 5, 'y' : 10}}, 'd': [1, 2, 3]}

def parse_dict(init, lkey=''):
    ret = {}
    for rkey,val in init.items():
        key = lkey+rkey
        if isinstance(val, dict):
            ret.update(parse_dict(val, key+'_'))
        else:
            ret[key] = val
    return ret

print(parse_dict(test,''))

Results:

$ python test.py
{'a': 1, 'c_a': 2, 'c_b_x': 5, 'd': [1, 2, 3], 'c_b_y': 10}

I am using python3.2, update for your version of python.

Answer 6

Simple function to flatten nested dictionaries. For Python 3, replace .iteritems() with .items()

def flatten_dict(init_dict):
    res_dict = {}
    if type(init_dict) is not dict:
        return res_dict

    for k, v in init_dict.iteritems():
        if type(v) == dict:
            res_dict.update(flatten_dict(v))
        else:
            res_dict[k] = v

    return res_dict

The idea/requirement was: Get flat dictionaries with no keeping parent keys.

Example of usage:

dd = {'a': 3, 
      'b': {'c': 4, 'd': 5}, 
      'e': {'f': 
                 {'g': 1, 'h': 2}
           }, 
      'i': 9,
     }

flatten_dict(dd)

>> {'a': 3, 'c': 4, 'd': 5, 'g': 1, 'h': 2, 'i': 9}

Keeping parent keys is simple as well.