Efficient way to remove keys with empty strings from a dict

Question

I have a dict and would like to remove all the keys for which there are empty value strings.

metadata = {u\'Composite:PreviewImage\': u\'(Binary data 101973          


        

Answer 1

Some benchmarking:

1. List comprehension recreate dict

In [7]: %%timeit dic = {str(i):i for i in xrange(10)}; dic['10'] = None; dic['5'] = None
   ...: dic = {k: v for k, v in dic.items() if v is not None} 
   1000000 loops, best of 7: 375 ns per loop

2. List comprehension recreate dict using dict()

In [8]: %%timeit dic = {str(i):i for i in xrange(10)}; dic['10'] = None; dic['5'] = None
   ...: dic = dict((k, v) for k, v in dic.items() if v is not None)
1000000 loops, best of 7: 681 ns per loop

3. Loop and delete key if v is None

In [10]: %%timeit dic = {str(i):i for i in xrange(10)}; dic['10'] = None; dic['5'] = None
    ...: for k, v in dic.items():
    ...:   if v is None:
    ...:     del dic[k]
    ...: 
10000000 loops, best of 7: 160 ns per loop

so loop and delete is the fastest at 160ns, list comprehension is half as slow at ~375ns and with a call to dict() is half as slow again ~680ns.

Wrapping 3 into a function brings it back down again to about 275ns. Also for me PyPy was about twice as fast as neet python.

Answer 2

It can get even shorter than BrenBarn's solution (and more readable I think)

{k: v for k, v in metadata.items() if v}

Tested with Python 2.7.3.

Answer 3

Quick Answer (TL;DR)

Example01

### example01 -------------------

mydict  =   { "alpha":0,
              "bravo":"0",
              "charlie":"three",
              "delta":[],
              "echo":False,
              "foxy":"False",
              "golf":"",
              "hotel":"   ",                        
            }
newdict =   dict([(vkey, vdata) for vkey, vdata in mydict.iteritems() if(vdata) ])
print newdict

### result01 -------------------
result01 ='''
{'foxy': 'False', 'charlie': 'three', 'bravo': '0'}
'''

Detailed Answer

Problem

• Context: Python 2.x
• Scenario: Developer wishes modify a dictionary to exclude blank values
  • aka remove empty values from a dictionary
  • aka delete keys with blank values
  • aka filter dictionary for non-blank values over each key-value pair

Solution

• example01 use python list-comprehension syntax with simple conditional to remove "empty" values

Pitfalls

• example01 only operates on a copy of the original dictionary (does not modify in place)
• example01 may produce unexpected results depending on what developer means by "empty"
  • Does developer mean to keep values that are falsy?
  • If the values in the dictionary are not gauranteed to be strings, developer may have unexpected data loss.
  • result01 shows that only three key-value pairs were preserved from the original set

Alternate example

• example02 helps deal with potential pitfalls
• The approach is to use a more precise definition of "empty" by changing the conditional.
• Here we only want to filter out values that evaluate to blank strings.
• Here we also use .strip() to filter out values that consist of only whitespace.

Example02

### example02 -------------------

mydict  =   { "alpha":0,
              "bravo":"0",
              "charlie":"three",
              "delta":[],
              "echo":False,
              "foxy":"False",
              "golf":"",
              "hotel":"   ",
            }
newdict =   dict([(vkey, vdata) for vkey, vdata in mydict.iteritems() if(str(vdata).strip()) ])
print newdict

### result02 -------------------
result02 ='''
{'alpha': 0,
  'bravo': '0', 
  'charlie': 'three', 
  'delta': [],
  'echo': False,
  'foxy': 'False'
  }
'''

See also

• list-comprehension
• falsy
• checking for empty string
• modifying original dictionary in place
• dictionary comprehensions
• pitfalls of checking for empty string

Answer 4

Python 2.X

dict((k, v) for k, v in metadata.iteritems() if v)

---

Python 2.7 - 3.X

{k: v for k, v in metadata.items() if v is not None}

Note that all of your keys have values. It's just that some of those values are the empty string. There's no such thing as a key in a dict without a value; if it didn't have a value, it wouldn't be in the dict.

Answer 5

Building on the answers from patriciasz and nneonneo, and accounting for the possibility that you might want to delete keys that have only certain falsy things (e.g. '') but not others (e.g. 0), or perhaps you even want to include some truthy things (e.g. 'SPAM'), then you could make a highly specific hitlist:

unwanted = ['', u'', None, False, [], 'SPAM']

Unfortunately, this doesn't quite work, because for example 0 in unwanted evaluates to True. We need to discriminate between 0 and other falsy things, so we have to use is:

any([0 is i for i in unwanted])

...evaluates to False.

Now use it to del the unwanted things:

unwanted_keys = [k for k, v in metadata.items() if any([v is i for i in unwanted])]
for k in unwanted_keys: del metadata[k]

If you want a new dictionary, instead of modifying metadata in place:

newdict = {k: v for k, v in metadata.items() if not any([v is i for i in unwanted])}

Answer 6

Dicts mixed with Arrays

• The answer at Attempt 3: Just Right (so far) from BlissRage's answer does not properly handle arrays elements. I'm including a patch in case anyone needs it. The method is handles list with the statement block of if isinstance(v, list):, which scrubs the list using the original scrub_dict(d) implementation.

    @staticmethod
    def scrub_dict(d):
        new_dict = {}
        for k, v in d.items():
            if isinstance(v, dict):
                v = scrub_dict(v)
            if isinstance(v, list):
                v = scrub_list(v)
            if not v in (u'', None, {}):
                new_dict[k] = v
        return new_dict

    @staticmethod
    def scrub_list(d):
        scrubbed_list = []
        for i in d:
            if isinstance(i, dict):
                i = scrub_dict(i)
            scrubbed_list.append(i)
        return scrubbed_list