Simultaneuos HTTP requests in PHP with cURL / Stoyan's phpied.com

The basic idea of a Web 2.0-style "mashup" is that you consume data from several services, often from different providers and combine them in interesting ways. This means you often need to do more than one HTTP request to a service or services. In PHP if you use something like file_get_contents() this means all the requests will be synchronous: a new one is fired only after the previous has completed. If you need to make three HTTP requests and each call takes a second, your app is delayed at least three seconds.

Solution

An improvement of course is to cache responses as much as possible, but at one point or another you still need to make those requests.

Using the curl_multi* family of cURL functions you can make those requests simultaneously. This way your app is as slow as the slowest request, as opposed to the sum of all requests. And that's something.

A function

Here's a little function I coded that will allow you do multi requests.

<?php
 
function multiRequest($data, $options = array()) {
 
  // array of curl handles
  $curly = array();
  // data to be returned
  $result = array();
 
  // multi handle
  $mh = curl_multi_init();
 
  // loop through $data and create curl handles
  // then add them to the multi-handle
  foreach ($data as $id => $d) {
 
    $curly[$id] = curl_init();
 
    $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
    curl_setopt($curly[$id], CURLOPT_URL,            $url);
    curl_setopt($curly[$id], CURLOPT_HEADER,         0);
    curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
 
    // post?
    if (is_array($d)) {
      if (!empty($d['post'])) {
        curl_setopt($curly[$id], CURLOPT_POST,       1);
        curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
      }
    }
 
    // extra options?
    if (!empty($options)) {
      curl_setopt_array($curly[$id], $options);
    }
 
    curl_multi_add_handle($mh, $curly[$id]);
  }
 
  // execute the handles
  $running = null;
  do {
    curl_multi_exec($mh, $running);
  } while($running > 0);
 
 
  // get content and remove handles
  foreach($curly as $id => $c) {
    $result[$id] = curl_multi_getcontent($c);
    curl_multi_remove_handle($mh, $c);
  }
 
  // all done
  curl_multi_close($mh);
 
  return $result;
}
 
?>

Consuming

The function accepts an array of URLs to hit and optionally an array of cURL options if you need to pass any. The first array can be a simple indexed array or URLs or it can be an array of arrays where the second has a key named "url". If you use the second way and you also have a key called "post", the function will do a post request.

The function returns an array of responses as strings. The keys in the result array match the keys in the input.

A GET example

Let's say you want to use some Yahoo search web services (consult YDN) to create a music artist band-o-pedia kind of mashup. Here's how you can search audio, video and images at the same time:

<?php
 
$data = array(
  'http://search.yahooapis.com/VideoSearchService/V1/videoSearch?appid=YahooDemo&query=Pearl+Jam&output=json',
  'http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Pearl+Jam&output=json',
  'http://search.yahooapis.com/AudioSearchService/V1/artistSearch?appid=YahooDemo&artist=Pearl+Jam&output=json'
);
$r = multiRequest($data);
 
echo '<pre>';
print_r($r);
 
?>

This will print something like:

Array
(
    [0] => {"ResultSet":{"totalResultsAvailable":"633","totalResultsReturned":...
    [1] => {"ResultSet":{"totalResultsAvailable":"105342","totalResultsReturned":...
    [2] => {"ResultSet":{"totalResultsAvailable":10,"totalResultsReturned":...
)

A POST example

There's an interesting Yahoo search service called term extraction which analyses content. It accepts POST requests. Here's how to consume this service with the function above, making two simultaneous requests:

<?php
$data = array(array(),array());
 
$data[0]['url']  = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction';
$data[0]['post'] = array();
$data[0]['post']['appid']   = 'YahooDemo';
$data[0]['post']['output']  = 'php';
$data[0]['post']['context'] = 'Now I lay me down to sleep,
                               I pray the Lord my soul to keep;
                               And if I die before I wake,
                               I pray the Lord my soul to take.';
 
 
$data[1]['url']  = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction';
$data[1]['post'] = array();
$data[1]['post']['appid']   = 'YahooDemo';
$data[1]['post']['output']  = 'php';
$data[1]['post']['context'] = 'Now I lay me down to sleep,
                               I pray the funk will make me freak;
                               If I should die before I waked,
                               Allow me Lord to rock out naked.';
 
$r = multiRequest($data);
 
print_r($r);
?>

And the result:

Array
(
    [0] => a:1:{s:9:"ResultSet";a:1:{s:6:"Result";s:5:"sleep";}}
    [1] => a:1:{s:9:"ResultSet";a:1:{s:6:"Result";a:3:{i:0;s:5:"freak";i:1;s:5:"sleep";i:2;s:4:"funk";}}}
)

Comments? Feedback? Find me on Twitter, Mastodon, Bluesky, LinkedIn, Threads