Split big files using PHP

Question

I want to split huge files (to be specific, tar.gz files) in multiple part from php code. Main reason to do this is, php\'s 2gb limit on 32bit system.

SO I want to s

Answer 1

PHP itself might not be able to... If you can figure out how to do this from your computers' command line, You should be able to then execute these commands using exec();

Answer 2

Splits are named as filename.part0 filename.part1 ...

<?php
function fsplit($file,$buffer=1024){
    //open file to read
    $file_handle = fopen($file,'r');
    //get file size
    $file_size = filesize($file);
    //no of parts to split
    $parts = $file_size / $buffer;

    //store all the file names
    $file_parts = array();

    //path to write the final files
    $store_path = "splits/";

    //name of input file
    $file_name = basename($file);

    for($i=0;$i<$parts;$i++){
        //read buffer sized amount from file
        $file_part = fread($file_handle, $buffer);
        //the filename of the part
        $file_part_path = $store_path.$file_name.".part$i";
        //open the new file [create it] to write
        $file_new = fopen($file_part_path,'w+');
        //write the part of file
        fwrite($file_new, $file_part);
        //add the name of the file to part list [optional]
        array_push($file_parts, $file_part_path);
        //close the part file handle
        fclose($file_new);
    }    
    //close the main file handle

    fclose($file_handle);
    return $file_parts;
}
?>

Answer 3

A simple method (if using Linux based server) is to use the exec command and to run the split command:

exec('split Large.tar.gz -b 4096k SmallParts'); // 4MB parts
/*    |     |            |      | |
      |     |            |______| |
      App   |                 |   |_____________
            The source file   |                 |
                              The split size    Out Filename
*/

See here for more details: http://www.computerhope.com/unix/usplit.htm

Or you can use: http://www.computerhope.com/unix/ucsplit.htm

exec('csplit -k -s -f part_ -n 3 LargeFile.tar.gz');

---

PHP runs within a single thread and the only way to increase this thread count is to create child process using the fork commands.

This is not resource friendly. What I would suggest is to look into a language that can do this fast and effectively. I would suggest using node.js.

Just install node on the server and then create a small script, called node_split for instance, that can do the job on its own for you.

But I do strongly advise that you do not use PHP for this job but use exec to allow the host operating system to do this.

Answer 4

$handle = fopen('source/file/path','r'); 
        $f = 1; //new file number
        while(!feof($handle))
        {
            $newfile = fopen('newfile/path/'.$f.'.txt','w'); //create new file to write to with file number
            for($i = 1; $i <= 5000; $i++) //for 5000 lines
            {
                $import = fgets($handle);
                //print_r($import);
                fwrite($newfile,$import);
                if(feof($handle))
                {break;} //If file ends, break loop
            }
            fclose($newfile);
            $f++; //Increment newfile number
        }
        fclose($handle);

Answer 5

• If you want to split files which are already on server, you can do it (simply use the file functions fread, fopen, fwrite, fseek to read/write part of the file).
• If you want to split files which are uploaded from the client, I am afraid you cannot.

Answer 6

function split_file($source, $targetpath='/split/', $lines=1000){

    $i=0;
    $j=1;
    $date = date("m-d-y");
    $buffer='';

    $handle = fopen ($_SERVER['DOCUMENT_ROOT'].$source, "r");

    while (!feof ($handle)) {
        $buffer .= fgets($handle, 4096);
        $i++;
        if ($i >= $lines) {
            $fname = $_SERVER['DOCUMENT_ROOT'].$targetpath."part_".$date.$j.".txt";

                 $fhandle = fopen($fname, "w") or die($php_errormsg);

            if (!$fhandle) {
                echo "Cannot open file ($fname)";
                //exit;
            }


            if (!fwrite($fhandle, $buffer)) {
                echo "Cannot write to file ($fname)";
                //exit;
            }
            fclose($fhandle);
            $j++;
            $buffer='';
            $i=0;
            $line+=10; // add 10 to $lines after each iteration. Modify this line as required
        }
    }
    fclose ($handle);
}