How to loop until EOF in Python?

Question

I need to loop until I hit the end of a file-like object, but I\'m not finding an \"obvious way to do it\", which makes me suspect I\'m overlooking something, well, obvious.

Answer 1

Have you seen how to iterate over lines in a text file?

for line in file_obj:
  use(line)

You can do the same thing with your own generator:

def read_blocks(file_obj, size):
  while True:
    data = file_obj.read(size)
    if not data:
      break
    yield data

for block in read_blocks(file_obj, 4):
  use(block)

See also:

• file.read

Answer 2

I see, as predicted, that the typical and most popular answer are using very specialized generators to "read 4 bytes at a time". Sometimes generality isn't any harder (and much more rewarding;-), so, I've suggested instead the following very general solution:

import operator
def funlooper(afun, *a, **k):
  wearedone = k.pop('wearedone', operator.not_)
  while True:
    data = afun(*a, **k)
    if wearedone(data): break
    yield data

Now your desired loop header is just: for len_name in funlooper(data.read, 4):.

Edit: made much more general by the wearedone idiom since a comment accused my slightly less general previous version (hardcoding the exit test as if not data:) of having "a hidden dependency", of all things!-)

The usual swiss army knife of looping, itertools, is fine too, of course, as usual:

import itertools as it

for len_name in it.takewhile(bool, it.imap(data.read, it.repeat(4))): ...

or, quite equivalently:

import itertools as it

def loop(pred, fun, *args):
  return it.takewhile(pred, it.starmap(fun, it.repeat(args)))

for len_name in loop(bool, data.read, 4): ...

Answer 3

I'd go with Tendayi's suggestion re function and iterator for readability:

def read4():
    len_name = data.read(4)
    if len_name:
        len_name = struct.unpack("<I", len_name)[0]
        return data.read(len_name)
    else:
        raise StopIteration

for d in iter(read4, ''):
    names.append(d)

Answer 4

You can combine iteration through iter() with a sentinel:

for block in iter(lambda: file_obj.read(4), ""):
  use(block)

Answer 5

I prefer the already mentioned iterator-based solution to turn this into a for-loop. Another solution written directly is Knuth's "loop-and-a-half"

while 1:
    len_name = data.read(4)
    if not len_name:
        break
    names.append(data.read(len_name))

You can see by comparison how that's easily hoisted into its own generator and used as a for-loop.

Answer 6

The EOF marker in python is an empty string so what you have is pretty close to the best you are going to get without writing a function to wrap this up in an iterator. I could be written in a little more pythonic way by changing the while like:

while len_name:
    len_name = struct.unpack("<I", len_name)[0]
    names.append(data.read(len_name))
    len_name = data.read(4)