json.loads allows duplicate keys in a dictionary, overwriting the first value

Question

>>> raw_post_data = request.raw_post_data
>>> print raw_post_data
{\"group\":{\"groupId\":\"2\", \"groupName\":\"GroupName\"}, \"members\":{\"1\":{         


        

Answer 1

Alternatively if you want to catch all the duplicate keys (per level) you can use a collections.Counter

from collections import Counter

class KeyWatcher(dict):

    def __init__(self, *args):
        duplicates = [d for d,i in Counter([pair[0] for pair in args[0]]).items() if i > 0]
        if duplicates:
            raise KeyError("Can't add duplicate keys {} to a json message".format(duplicates))
        self.update(*args[0])

json.loads(raw_post_data, object_pairs_hook=KeyWatcher)

Answer 2

One alternative I wrote based on the solution posted by other users of this question is to convert those duplicates into an array:

def array_on_duplicate_keys(ordered_pairs):
    """Convert duplicate keys to arrays."""
    d = {}
    for k, v in ordered_pairs:
        if k in d:
            if type(d[k]) is list:
                d[k].append(v)
            else:
                d[k] = [d[k],v]
        else:
           d[k] = v
    return d

And then:

dict = json.loads('{"x": 1, "x": 2}', object_pairs_hook=array_on_duplicate_keys)

gives you the output:

{'x': [1, 2]}

Later one, one can check easily how meany duplicates an entry has by using:

if type(dict['x']) is list:
    print('Non-unique entry in dict at x, found', len(dict['x']),'repetitions.')

Answer 3

The rfc 4627 for application/json media type recommends unique keys but it doesn't forbid them explicitly:

The names within an object SHOULD be unique.

From rfc 2119:

SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.

import json

def dict_raise_on_duplicates(ordered_pairs):
    """Reject duplicate keys."""
    d = {}
    for k, v in ordered_pairs:
        if k in d:
           raise ValueError("duplicate key: %r" % (k,))
        else:
           d[k] = v
    return d

json.loads(raw_post_data, object_pairs_hook=dict_raise_on_duplicates)
# -> ValueError: duplicate key: u'1'

Answer 4

This is a linter-fixed and type-annotated version of the answer by jfs. Issues highlighted by various linters were addressed. It is also modernized for Python 3.6+ to use f-strings.

import json
from typing import Any, Dict, Hashable, List, Tuple

def raise_on_duplicate_keys(ordered_pairs: List[Tuple[Hashable, Any]]) -> Dict:
    """Raise ValueError if a duplicate key exists in provided ordered list of pairs, otherwise return a dict."""
    dict_out = {}
    for key, val in ordered_pairs:
        if key in dict_out:
            raise ValueError(f'Duplicate key: {key}')
        else:
            dict_out[key] = val
    return dict_out

json.loads('{"x": 1, "x": 2}', object_pairs_hook=raise_on_duplicate_keys)

ordered_pairs above is a list of tuples, with each tuple having a key and a value. Refer to the docs for object_pairs_hook.