Compute average and standard deviation with awk

Question

I have a \'file.dat\' with 24 (rows) x 16 (columns) data.

I have already tested the following awk script that computes de average of each column.

t         


        

Answer 1

Standard deviation is

stdev = sqrt((1/N)*(sum of (value - mean)^2))

But there is another form of the formula which does not require you to know the mean beforehand. It is:

stdev = sqrt((1/N)*((sum of squares) - (((sum)^2)/N)))

(A quick web search for "sum of squares" formula for standard deviation will give you the derivation if you are interested)

To use this formula, you need to keep track of both the sum and the sum of squares of the values. So your awk script will change to:

    awk '{for(i=1;i<=NF;i++) {sum[i] += $i; sumsq[i] += ($i)^2}} 
          END {for (i=1;i<=NF;i++) {
          printf "%f %f \n", sum[i]/NR, sqrt((sumsq[i]-sum[i]^2/NR)/NR)}
         }' file.dat >> aver-std.dat