How to solve memory error in mtrand.RandomState.choice?

Question

I\'m trying to sample 1e7 items from 1e5 strings but getting a memory error. It\'s fine sampling 1e6 items from 1e4 strings. I\'m on a 64bit machine with 4GB RAM and don\'t

Answer 1

You can work round this using a generator function:

def item():
    for i in xrange(N):
      yield "id%010d"%np.random.choice(N//K,1)

This avoids needing all the items in memory at once.