Is making asynchronous HTTP requests possible with PHP?

Question

I have a PHP script that needs to download several files from a remote server. At the moment I just have a loop downloading and processing the files with cURL, which means t

Answer 1

The library of @stil is so, so cool. Many thanks to him!

Still, I have written nice utility function that makes it very easy to get asynchronously content from multiple URLs (APIs in my case) and to return them without losing information which is which.

You simply run it by passing key => value array as an input and it returns key => response array as result :- )

/** 
     * This function runs multiple GET requests parallely.<br />
     * @param array $urlsArray needs to be in format:<br />
     * <i>array(<br />
     * [url_unique_id_1] => [url_for_request_1],<br />
     * [url_unique_id_2] => [url_for_request_2],<br />
     * [url_unique_id_3] => [url_for_request_3]<br />
     * )</i><br />
     * e.g. input like:<br />
     *  <i>array(<br />
     * &nbsp; "myemail@gmail.com" =>
     * &nbsp; "http://someapi.com/results?search=easylife",<br />
     * &nbsp; "michael@gmail.com" =>
     * &nbsp; "http://someapi.com/results?search=safelife"<br />
     * )</i>
     * @return array An array where for every <i>url_unique_id</i> response to this request 
     * is returned e.g.<br />
     * <i>array(<br />
     * &nbsp; "myemail@gmail.com" => <br />
     * &nbsp; "Work less, enjoy more",<br />
     * &nbsp; "michael@gmail.com" => <br />
     * &nbsp; "Study, work, pay taxes"<br />
     * )</i>
     *  */
    public function getResponsesFromUrlsAsynchronously(array $urlsArray, $timeout = 8) {
        $queue = new \cURL\RequestsQueue;

        // Set default options for all requests in queue
        $queue->getDefaultOptions()
                ->set(CURLOPT_TIMEOUT, $timeout)
                ->set(CURLOPT_RETURNTRANSFER, true);

        // =========================================================================
        // Define some extra variables to be used in callback

        global $requestUidToUserUrlIdentifiers;
        $requestUidToUserUrlIdentifiers = array();

        global $userIdentifiersToResponses;
        $userIdentifiersToResponses = array();

        // =========================================================================

        // Set function to be executed when request will be completed
        $queue->addListener('complete', function (\cURL\Event $event) {

            // Define user identifier for this url
            global $requestUidToUserUrlIdentifiers;
            $requestId = $event->request->getUID();
            $userIdentifier = $requestUidToUserUrlIdentifiers[$requestId];

            // =========================================================================

            $response = $event->response;
            $json = $response->getContent(); // Returns content of response

            $apiResponseAsArray = json_decode($json, true);
            $apiResponseAsArray = $apiResponseAsArray['jobs'];

            // =========================================================================
            // Store this response in proper structure
            global $userIdentifiersToResponses;
            $userIdentifiersToResponses[$userIdentifier] = $apiResponseAsArray;
        });

        // =========================================================================

        // Add all request to queue
        foreach ($urlsArray as $userUrlIdentifier => $url) {
            $request = new \cURL\Request($url);
            $requestUidToUserUrlIdentifiers[$request->getUID()] = $userUrlIdentifier;
            $queue->attach($request);
        }

        // =========================================================================

        // Execute queue
        $queue->send();

        // =========================================================================

        return $userIdentifiersToResponses;
    }

Answer 2

In PHP 7.0 & Apache 2.0, as said in PHP exec Document, redirecting the output, by adding " &> /dev/null &" at the end of the command, could makes it running on background, just remember to wrap the command correctly.

$time = microtime(true);
$command = '/usr/bin/curl -H \'Content-Type: application/json\' -d \'' . $curlPost . '\' --url \'' . $wholeUrl . '\' >> /dev/shm/request.log 2> /dev/null &';
exec($command);
echo (microtime(true) - $time) * 1000 . ' ms';

Above works well for me, takes only 3ms, but following won't work, takes 1500ms.

$time = microtime(true);
$command = '/usr/bin/curl -H \'Content-Type: application/json\' -d \'' . $curlPost . '\' --url ' . $wholeUrl;
exec($command . ' >> /dev/shm/request.log 2> /dev/null &');
echo (microtime(true) - $time) * 1000 . ' ms';

In total, adding " &> /dev/null &" at the end of your command may helps, just remember to WRAP your command properly.

Answer 3

Yes.

There is the multirequest PHP library (or see: archived Google Code project). It's a multi-threaded CURL library.

As another solution, you could write a script that does that in a language that supports threading, like Ruby or Python. Then, just call the script with PHP. Seems rather simple.

Answer 4

For PHP5.5+, mpyw/co is the ultimate solution. It works as if it is tj/co in JavaScript.

Example

Assume that you want to download specified multiple GitHub users' avatars. The following steps are required for each user.

1. Get content of http://github.com/mpyw (GET HTML)
2. Find <img class="avatar" src="..."> and request it (GET IMAGE)

---: Waiting my response
...: Waiting other response in parallel flows

Many famous curl_multi based scripts already provide us the following flows.

        /-----------GET HTML\  /--GET IMAGE.........\
       /                     \/                      \ 
[Start] GET HTML..............----------------GET IMAGE [Finish]
       \                     /\                      /
        \-----GET HTML....../  \-----GET IMAGE....../

However, this is not efficient enough. Do you want to reduce worthless waiting times ...?

        /-----------GET HTML--GET IMAGE\
       /                                \            
[Start] GET HTML----------------GET IMAGE [Finish]
       \                                /
        \-----GET HTML-----GET IMAGE.../

Yes, it's very easy with mpyw/co. For more details, visit the repository page.

Answer 5

Check out curl-easy. It supports both blocking and not-blocking requests in parallel or single request at once. Also, it is unit-tested, unlike many simple or buggy libraries.

Disclosure: I am the author of this library. The library has it's own test suite so I'm pretty confident it is robust.

Also, check out example of use below:

<?php
// We will download info about 2 YouTube videos:
// http://youtu.be/XmSdTa9kaiQ and
// http://youtu.be/6dC-sm5SWiU

// Init queue of requests
$queue = new cURL\RequestsQueue;
// Set default options for all requests in queue
$queue->getDefaultOptions()
    ->set(CURLOPT_TIMEOUT, 5)
    ->set(CURLOPT_RETURNTRANSFER, true);
// Set callback function to be executed when request will be completed
$queue->addListener('complete', function (cURL\Event $event) {
    $response = $event->response;
    $json = $response->getContent(); // Returns content of response
    $feed = json_decode($json, true);
    echo $feed['entry']['title']['$t'] . "\n";
});

$request = new cURL\Request('http://gdata.youtube.com/feeds/api/videos/XmSdTa9kaiQ?v=2&alt=json');
$queue->attach($request);

$request = new cURL\Request('http://gdata.youtube.com/feeds/api/videos/6dC-sm5SWiU?v=2&alt=json');
$queue->attach($request);

// Execute queue
$queue->send();