Can I store an iterator in a file which I can read from later? Will this reduce space consumption? [closed]

问题

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.

Closed 7 years ago.

Let's say I have a very large integer, around the order of 10**200. Now storing the integer in a file will take some amount of space.

If I convert it into an iterator using yield, can I store the iterator in a file instead? Will this save any resources?

The iterator can be generated like this:

def rec():
  for i in range(0,10**200):
    yield i

iterable = rec()

回答1:

Building on larsmans answer, a custom iterator can be built to do this:

class my_large_num(object):

    def __init__(self):
        self.num_iterations = 0

    def __iter__(self):
        return self


    def next(self):
        if self.num_iterations < 1:
            self.num_iterations += 1
            return 10**200
        else:
            raise StopIteration()

You can then:

import pickle
pickled_repr = pickle.dumps(my_large_num())
restored_object = pickle.loads(pickled_repr)
sum(restored_object)

This works because underneath, iterable objects have a next() function which raises StopIteration when done. All we're doing is creating a class that implements this functionality.

In this specific case, regardless of the fact you have stored the class in a file, you still need to perform the iteration, and thus store 10**200 in memory, so you gain no functionality except generating the number on demand, which you can do without serializing the object.

You might be thinking of mmap style space saving. This maps memory to a file - note however this still affects the usable memory of your program.

回答2:

I'm assuming this is what you'd like to do:

def f():
    yield 10**200

Then save f() in a file. The answer is no, that won't work. An generator like f() (note: that's generator, not iterable) cannot be pickled or otherwise serialized unless you turn it into a custom iterator with special-purpose pickling support.

回答3:

You can use the Shelve Module to store this.

A “shelf” is a persistent, dictionary-like object. The difference with “dbm” databases is that the values (not the keys!) in a shelf can be essentially arbitrary Python objects — anything that the pickle module can handle.

The following types can be pickled

回答4:

An integer of the value 10**200 does not take a large amount of space. Encoded in base 10 ASCII, that takes only 201 characters. If you're willing to store your data in binary, then you're looking at only 85ish

---

If you mean "iterable", that doesn't make much sense either - an iterable is essentially a function, and you already have the function saved - it's in the source file.

来源：https://stackoverflow.com/questions/13220488/can-i-store-an-iterator-in-a-file-which-i-can-read-from-later-will-this-reduce