I need to operate in big 3-dim non-sparse matrices in Matlab. Using pure vectorization gives a high computation time. So, I have tried to split the operations into 10 blocks and then parse the results. I got surprised when I saw the the pure vectorization does not scale very well with the data size as presented in the following figure.
I include an example of the two approaches.
% Parameters:
M = 1e6; N = 50; L = 4; K = 10;
% Method 1: Pure vectorization
mat1 = randi(L,[M,N,L]);
mat2 = repmat(permute(1:L,[3 1 2]),M,N);
result1 = nnz(mat1>mat2)./(M+N+L);
% Method 2: Split computations
result2 = 0;
for ii=1:K
mat1 = randi(L,[M/K,N,L]);
mat2 = repmat(permute(1:L,[3 1 2]),M/K,N);
result2 = result2 + nnz(mat1>mat2);
end
result2 = result2/(M+N+L);
Hence, I wonder if there is any other approach that makes big matrix operations in Matlab more efficient. I know it is a quite broad question, but I will take the risk :)
Edit:
Using the implementation of @Shai
% Method 3
mat3 = randi(L,[M,N,L]);
result3 = nnz(bsxfun( @gt, mat3, permute( 1:L, [3 1 2] ) ))./(M+N+L);
The times are:
Why repmat
and not bsxfun
?
result = nnz(bsxfun( @gt, mat1, permute( 1:L, [3 1 2] ) ))./(M+N+L);
It seems like you are using up your RAM and the OS starts to allocate room in swap for the very large matrics. Memory swapping is always a very time consuming operation and it gets worse as the amount of memory you require increases.
I believe you are witnessing thrashing.
来源:https://stackoverflow.com/questions/22383813/efficient-operations-of-big-non-sparse-matrices-in-matlab