Remove duplicate dict in list in Python

Question

I have a list of dicts, and I\'d like to remove the dicts with identical key and value pairs.

For this list: [{\'a\': 123}, {\'b\': 123}, {\'a\': 123}]<

Answer 1

If you are using Pandas in your workflow, one option is to feed a list of dictionaries directly to the pd.DataFrame constructor. Then use drop_duplicates and to_dict methods for the required result.

import pandas as pd

d = [{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}, {'a': 123, 'b': 1234}]

d_unique = pd.DataFrame(d).drop_duplicates().to_dict('records')

print(d_unique)

[{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}]