Download List of Files Simultaneously in PHP

Question

With PHP I create an array of a CSV file containing URLs to files I want to download with this line:

$urls = explode(\',\',file_get_contents(\'urls.csv\'));
         


        

Answer 1

If you have access to curl you can use curl_multi_exec. First chunk your $urls array into groups of how many you want to execute simultaneously, then process each group using curl_multi_exec.

$all_urls = ['http://www.google.com',
'http://www.yahoo.com',
'http://www.bing.com',
'http://www.twitter.com',
'http://www.wikipedia.org',
'http://www.stackoverflow.com'];

$chunked_urls = array_chunk($all_urls,3); //chunk into groups of 3

foreach($chunked_urls as $i => $urls) {

    $handles = [];    
    $mh = curl_multi_init();  

    foreach($urls as $url) {
        $ch = curl_init($url);  
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_multi_add_handle($mh, $ch);
        $handles[] = $ch;
    }

    // execute all queries simultaneously, and continue when all are complete
    $running = null;
    do {
        curl_multi_exec($mh, $running);
    } while ($running);

    foreach($handles as $handle) {
        file_put_contents("/tmp/output",curl_multi_getcontent($handle),FILE_APPEND);
        curl_multi_remove_handle($mh, $handle);        
    }

    curl_multi_close($mh);

    print "Finished chunk $i\n";
}