matrix calculation error

Question

I am using R tool to calculate SVD (svd(m)) and it works on small matrix but as I pass it 20Kx20X matrix. After processing, it gives the following erro

Answer 1

If you're storing doubles that are 8 bytes, 20Kx20K means 8*20,000*20,000/1024/1024 ~ 3GB of RAM to hold the whole thing in memory.

I don't know how much RAM you've got available, but I'd look into what R can do to serialize the matrix out to disk as needed.

Is the matrix sparse or banded? Can you do something to decrease the amount of memory you need?

How large is the null space for you matrix? What's the condition number (ratio of largest-to-smallest eigenvalue)? A large condition number can be an indication of difficulties in solving. A matrix need not have a zero row or column to be nearly singular.

UPDATE:

Based on your comment, I'd say that RAM is the least of your problems. Sounds like it's possible to hold the entire matrix in memory - if you can address it all. You can address the entire matrix. You're running on a 64-bit OS - is your version of R 64-bit as well?

Unfortunately, one of the byproducts of SVD is to get the size of the null space.

You can get the minimum eigenvalue for your matrix using Jacobi iteration. Lanczos might be a good choice for getting the maximum eigenvalue. It'd be a lot of work to get all of them; you might just want the five smallest and largest to assess.

Anytime I experience an error with some software I immediately paste it into a Google search. At least it's comforting to know that I'm not the first to experience a particular problem:

http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=Error+in+svd(m)+:+infinite+or+missing+values+in+'x'

Answer 2

Possibly the svd calculation itself also uses a lot of memory. If we compare to MATLAB, we see that the svd calculation allocates just as much memory as the matrix itself uses, so if you already ise 3GB of memory, the svd calculation possibly allocates another 3GB, which gives 6GB of memory.

Answer 3

I am guessing that your problem is not related to memory size, although I am not able to process a 20Kx20K matrix on my 4GB memory machine.

The reason for this guess is that the first line of code inside svd() is the following:

if (any(!is.finite(x))) 
    stop("infinite or missing values in 'x'")

In other words, the svd() function test first whether there are any infinite values in your data.

This happens before any further processing. So, if you had memory problems, these would be apparent even before your call to svd().

I suggest you check for infinite values:

x <- c(0, Inf, NA, NULL)
which(!is.finite(x))

[1] 2 3

This indicates that the second and third values are considered to be not finite. In other words, any NA values in your data will cause your error.