Is there a way to remove nan from a dictionary filled with data?

Question

I have a dictionary that is filled with data from two files I imported, but some of the data comes out as nan. How do I remove the pieces of data with nan?

My code i

Answer 1

from math import isnan

if nans are being stored as keys:

# functional
clean_dict = filter(lambda k: not isnan(k), my_dict)

# dict comprehension
clean_dict = {k: my_dict[k] for k in my_dict if not isnan(k)}

if nans are being stored as values:

# functional
clean_dict = filter(lambda k: not isnan(my_dict[k]), my_dict)

# dict comprehension
clean_dict = {k: my_dict[k] for k in my_dict if not isnan(my_dict[k])}

Answer 2

Instead of trying to remove the NaNs from your dictionary, you should further investigate why NaNs are getting there in the first place.

It gets difficult to use NaNs in a dictionary, as a NaN does not equal itself.

Check this out for more information: NaNs as key in dictionaries

Answer 3

With simplejson

import simplejson

clean_dict  = simplejson.loads(simplejson.dumps(my_dict, ignore_nan=True))
## or depending on your needs
clean_dict  = simplejson.loads(simplejson.dumps(my_dict, allow_nan=False))

Answer 4

Know old, but here is what worked for me and simple - remove NaNs on reading of the CSV upfront:

orangebook = pd.read_csv('C:\Users\WEGWEIS_JAKE\Desktop\Work Programs\Code Files\products2.txt',sep='~', parse_dates=['Approval_Date']).dropna()

I also like to convert to dictionary at the same time:

orangebook = pd.read_csv('C:\Users\WEGWEIS_JAKE\Desktop\Work Programs\Code Files\products2.txt',sep='~', parse_dates=['Approval_Date']).dropna().to_dict()