How can I read large text files in Python, line by line, without loading it into memory?

Question

I need to read a large file, line by line. Lets say that file has more than 5GB and I need to read each line, but obviously I do not want to use readlines() bec

Answer 1

This might be useful when you want to work in parallel and read only chunks of data but keep it clean with new lines.

def readInChunks(fileObj, chunkSize=1024):
    while True:
        data = fileObj.read(chunkSize)
        if not data:
            break
        while data[-1:] != '\n':
            data+=fileObj.read(1)
        yield data