Distributed cross correlation matrix computation

Question

How can I calculate pearson cross correlation matrix of large (>10TB) data set, possibly in distributed manner ? Any efficient distributed algorithm suggestion will be ap

Answer 1

Each local data sets can converted into stdv and covariances. Also stdv and covariance and sum make correlation.

This is working example https://github.com/jeesim2/distributed-correlation

Answer 2

To start with, have a look at this to see if things are going right. You may then refer to any of these implementations: MPI/OpenMP: Agomezl or Meismyles, MapReduce: Vangjee or Seawolf42. It'd also be interesting to read this before you proceed. On a different note, James's thesis provides some pointers if you're interested in computing the correlations that are robust to outliers.