How to read a large file line by line?

Question

I want to read a file line by line, but without completely loading it in memory.

My file is too large to open in memory, and if try to do so I always get out of memo

Answer 1

You can use the fgets() function to read the file line by line:

$handle = fopen("inputfile.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        // process the line read.
    }

    fclose($handle);
} else {
    // error opening the file.
} 

Answer 2

If you're opening a big file, you probably want to use Generators alongside fgets() to avoid loading the whole file into memory:

/**
 * @return Generator
 */
$fileData = function() {
    $file = fopen(__DIR__ . '/file.txt', 'r');

    if (!$file)
        die('file does not exist or cannot be opened');

    while (($line = fgets($file)) !== false) {
        yield $line;
    }

    fclose($file);
};

Use it like this:

foreach ($fileData() as $line) {
    // $line contains current line
}

This way you can process individual file lines inside the foreach().

Note: Generators require >= PHP 5.5

Answer 3

Function to Read with array return

function read_file($filename = ''){
    $buffer = array();
    $source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
    while (!feof($source_file)) {
        $buffer[] = fread($source_file, 4096);  // use a buffer of 4KB
    }
    return $buffer;
}

Answer 4

Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.

Answer 5

SplFileObject is useful when it comes to dealing with large files.

function parse_file($filename)
{
    try {
        $file = new SplFileObject($filename);
    } catch (LogicException $exception) {
        die('SplFileObject : '.$exception->getMessage());
    }
    while ($file->valid()) {
        $line = $file->fgets();
        //do something with $line
    }

    //don't forget to free the file handle.
    $file = null;
}

Answer 6

This how I manage with very big file (tested with up to 100G). And it's faster than fgets()

$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if ($fh = fopen("file.txt", "r")) { 
    $left='';
    while (!feof($fh)) {// read the file
       $temp = fread($fh, $block);  
       $fgetslines = explode("\n",$temp);
       $fgetslines[0]=$left.$fgetslines[0];
       if(!feof($fh) )$left = array_pop($lines);           
       foreach ($fgetslines as $k => $line) {
           //do smth with $line
        }
     }
}
fclose($fh);