Linear regression in NumPy with very large matrices - how to save memory?

Question

So I have these ginormous matrices X and Y. X and Y both have 100 million rows, and X has 10 columns. I\'m trying to implement linear regression with these matrices, and I nee

Answer 1

RAM's pretty cheap - you should consider investing. A system with 24 Gig of RAM doesn't necessarily cost an arm and a leg anymore - one of Dell's lower-end servers can pack in that much.

If the matrices are sparse (lots of zeros), use a sparse matrix class to save a lot of RAM.

If the matrices aren't sparse, you'll either want more RAM (or at least more Virtual Memory), or to do your matrix operations using disk files.

Disk files are of course an order of magnitude slower than RAM, and thrashing your virtual memory system could actually be worse than that, depending on your access patterns.