Pandas : compute mean or std (standard deviation) over entire dataframe

Question

Here is my problem, I have a dataframe like this :

    Depr_1  Depr_2  Depr_3
S3  0   5   9
S2  4   11  8
S1  6   11  12
S5  0   4   11
S4  4   8   8
         


        

Answer 1

You could convert the dataframe to be a single column with stack (this changes the shape from 5x3 to 15x1) and then take the standard deviation:

df.stack().std()         # pandas default degrees of freedom is one

Alternatively, you can use values to convert from a pandas dataframe to a numpy array before taking the standard deviation:

df.values.std(ddof=1)    # numpy default degrees of freedom is zero

Unlike pandas, numpy will give the standard deviation of the entire array by default, so there is no need to reshape before taking the standard deviation.

A couple of additional notes:

• The numpy approach here is a bit faster than the pandas one, which is generally true when you have the option to accomplish the same thing with either numpy or pandas. The speed difference will depend on the size of your data, but numpy was roughly 10x faster when I tested a few different sized dataframes on my laptop (numpy version 1.15.4 and pandas version 0.23.4).

• The numpy and pandas approaches here will not give exactly the same answers, but will be extremely close (identical at several digits of precision). The discrepancy is due to slight differences in implementation behind the scenes that affect how the floating point values get rounded.