How to read big file in php without being memory limit

问题

I'm trying to read a file line by line. The problem is the file was too big(over 500000 line) and I reach out the memory limit. I wonder how to read the file without being memory limit.

I'm thinking about the solution multi threads(like split the file into smaller group(100000 line per group) and read it in multi threads), but I don't know how to do it in detail. Please help me(Sorry for bad English).

Here is my code

$fn = fopen("myfile.txt", "r");

while(!feof($fn)) {
    $result = fgets($fn);
    echo $result;
}

fclose($fn);

回答1:

You could use a generator to handle the memory usage. This is just an example written by a user on the documentation page:

function getLines($file)
{
    $f = fopen($file, 'r');

    try {
        while ($line = fgets($f)) {
            yield $line;
        }
    } finally {
        fclose($f);
    }
}

foreach (getLines("file.txt") as $n => $line) {
    // insert the line into db or do whatever you want with it.
}

A generator allows you to write code that uses foreach to iterate over a set of data without needing to build an array in memory, which may cause you to exceed a memory limit, or require a considerable amount of processing time to generate. Instead, you can write a generator function, which is the same as a normal function, except that instead of returning once, a generator can yield as many times as it needs to in order to provide the values to be iterated over.

回答2:

PHP cleans memory best when a scope is cleared in my experience. A loop doesn't count as a scope, but a function does.
So handing your file pointer to a function, doing your database things within the function and then exiting the function for the loop, where you can call gc_collect_cycles() should help with managing your memory and to force php to clean up after itself.

I also recommend turning off echo, but rather log to a file. You can then use a command tail -f filename to read that log output(windows linux subsystem, git for windows bash, or on linux)

I use a similar method to below to handle large files with millions of entries, and it helps with staying under the memory limit.

function dostuff($fn) 
{
    $result = fgets($fn);
    // store database, do transforms, whatever
    echo $result;
}

$fn = fopen("myfile.txt", "r");

while(!feof($fn)) {
    dostuff($fn);
    flush(); // only need this if you do the echo thing.
    gc_collect_cycles();
}

fclose($fn);

回答3:

You can use readfile and ob_get_level to manage memory and output buffering.

readfile() will not present any memory issues, even when sending large files, on its own. If you encounter an out of memory error ensure that output buffering is off with ob_get_level().

You may still have PHP output buffering active while performing the read.

Check that with:

You can use the following at the start of your script to stop output buffering if it's already started:

if (ob_get_level()) {
  ob_end_clean();
}

回答4:

You can set the memory limit like ini_set('memory_limit',-1);//You script will not stoped until its finished the reading. but this is wrong way beacuse its take your CPU utilization time on server.

Better is to divide the file in chunks,

covert you file data to array then you can easily read it in chunks like

$file_lines = file('mytext.txt');
foreach ($file_lines as $line) {
    echo $line;
}

$file_lines is your array.

来源：https://stackoverflow.com/questions/55457149/how-to-read-big-file-in-php-without-being-memory-limit

标签

php

file

memory-limit