Very large matrices using Python and NumPy

Question

NumPy is an extremely useful library, and from using it I\'ve found that it\'s capable of handling matrices which are quite large (10000 x 10000) easily, but begins to strug

Answer 1

Sometimes one simple solution is using a custom type for your matrix items. Based on the range of numbers you need, you can use a manual dtype and specially smaller for your items. Because Numpy considers the largest type for object by default this might be a helpful idea in many cases. Here is an example:

In [70]: a = np.arange(5)

In [71]: a[0].dtype
Out[71]: dtype('int64')

In [72]: a.nbytes
Out[72]: 40

In [73]: a = np.arange(0, 2, 0.5)

In [74]: a[0].dtype
Out[74]: dtype('float64')

In [75]: a.nbytes
Out[75]: 32

And with custom type:

In [80]: a = np.arange(5, dtype=np.int8)

In [81]: a.nbytes
Out[81]: 5

In [76]: a = np.arange(0, 2, 0.5, dtype=np.float16)

In [78]: a.nbytes
Out[78]: 8