Python list memory error

Question

I am trying to make a list with 2 raised to 30 elements but am getting a memory error. Why so? Is it outside the maximum limit of a list in python?

m=[None]*(2**         


        

Answer 1

You are trying to create a huge list here and you do not have enough free RAM to do this. How much RAM will you need?

It will be approximately 8Gb on my machine. You can find the size of None on your machine:

import sys
sys.getsizeof(None) # 16 here

But for a list you can approximate it this way:

sys.getsizeof([None] * (2**20)) # 8388680

and hoping that for 2**30 it will take approximately 2**10 times more you will end up with 8388680 * 2**10 which is approximately 8Gb.

Answer 2

Yes, there is a limit to how many elements a Python list can hold, see sys.maxsize. You did not hit it however; very few machines have enough memory to hold that many items.

You are trying to create a list with 1073741824 references; each reference takes memory too. It depends on your OS how much, but typically that is going to be 4 bytes for a 32-bit system, 8 bytes for a 64-bit OS, where 2^30 elements would take 4 GB or 8GB of memory, just for the list references.

4GB plus other elements is already easily more than what most current operating systems will permit a single process to use in memory.

On my Mac OS X machine (using 64-bit OS), sys.maxsize is 2^63, and Python object references in a list take 8 bytes:

>>> import sys
>>> sys.maxsize
9223372036854775807
>>> sys.maxsize.bit_length()
63
>>> sys.getsizeof([])  # empty list overhead
72
>>> sys.getsizeof([None]) - sys.getsizeof([])  # size of one reference
8

So to create a list with sys.maxsize elements you'd need 64 exbibytes of memory, just for the references. That is more than what a 64-bit computer could address (the practical maximum is about 16 exbibytes).

All this is ignoring the memory footprint that the objects you are referencing in the list will take. None is a singleton, so it'll only ever take a fixed amount of memory. But presumably you were going to store something else in that list, in which case you need to take that into account too.

And generally speaking you should never need to create such a large list. Use different techniques; create a sparse structure using a dictionary for example, I doubt you were planning to address all those 2^30 indexes directly in your algorithm.