'invalid value encountered in double_scalars' warning, possibly numpy

Question

As I run my code I get these warnings, always in groups of four, sporadically. I have tried to locate the source by placing debug messages before and after certain statement

Answer 1

I ran into similar problem - Invalid value encountered in ... After spending a lot of time trying to figure out what is causing this error I believe in my case it was due to NaN in my dataframe. Check out working with missing data in pandas.

None == None True

np.nan == np.nan False

When NaN is not equal to NaN then arithmetic operations like division and multiplication causes it throw this error.

Couple of things you can do to avoid this problem:

1. Use pd.set_option to set number of decimal to consider in your analysis so an infinitesmall number does not trigger similar problem - ('display.float_format', lambda x: '%.3f' % x).

2. Use df.round() to round the numbers so Panda drops the remaining digits from analysis. And most importantly,

3. Set NaN to zero df=df.fillna(0). Be careful if Filling NaN with zero does not apply to your data sets because this will treat the record as zero so N in the mean, std etc also changes.