Handle Large File with PHP

问题

I have a file with the size of around 10 GB or more. The file contains only numbers ranging from 1 to 10 on each line and nothing else. Now the task is to read the data[numbers] from the file and then sort the numbers in ascending or descending order and create a new file with the sorted numbers.

Can anyone of you please help me with the answer?

回答1:

I'm assuming this is somekind of homework and goal for this is to sort more data than you can hold in your RAM?

Since you only have numbers 1-10, this is not that complicated task. Just open your input file and count how many occourances of every specific number you have. After that you can construct simple loop and write values into another file. Following example is pretty self explainatory.

$inFile = '/path/to/input/file';
$outFile = '/path/to/output/file';
$input = fopen($inFile, 'r');
if ($input === false) {
    throw new Exception('Unable to open: ' . $inFile);
}
//$map will be array with size of 10, filled with 0-s
$map = array_fill(1, 10, 0);
//Read file line by line and count how many of each specific number you have
while (!feof($input)) {
    $int = (int) fgets($input);
    $map[$int]++;
}
fclose($input);
$output = fopen($outFile, 'w');
if ($output === false) {
    throw new Exception('Unable to open: ' . $outFile);
}
/*
 * Reverse array if you need to change direction between
 * ascending and descending order
 */
//$map = array_reverse($map);
//Write values into your output file
foreach ($map AS $number => $count) {
    $string = ((string) $number) . PHP_EOL;
    for ($i = 0; $i < $count; $i++) {
        fwrite($output, $string);
    }
}
fclose($output);

Taking into account the fact, that you are dealing with huge files, you should also check script execution time limit for your PHP environment, following example will take VERY long for 10GB+ sized files, but since I didn't see any limitations concerning execution time and performance in your question, I'm assuming it is OK.

回答2:

I had a similar issue before. Trying to manipulate such a large file ended up being huge drain on resources and it couldn't cope. The easiest solution I ended up with was to try and import it into a MySQL database using a fast data dump function called LOAD DATA INFILE

http://dev.mysql.com/doc/refman/5.1/en/load-data.html

Once it's in you should be able to manipulate the data.

Alternatively, you could just read the file line by line while outputting the result into another file line by line with the sorted numbers. Not too sure how well this would work though.

Have you had any previous attempts at it or are you just after a possible method of doing it?

回答3:

If that's all you don't need PHP (if you have a Linux maschine at hand):

sort -n file > file_sorted-asc
sort -nr file > file_sorted-desc

Edit: OK, here's your solution in PHP (if you have a Linux maschine at hand):

<?php

// Sort ascending
`sort -n file > file_sorted-asc`;

// Sort descending
`sort -nr file > file_sorted-desc`;

?>

来源：https://stackoverflow.com/questions/19515439/handle-large-file-with-php

标签

php

sorting

large-files

file-handling