Facebook mutual friends and FQL 4999/5000 record limit

Question

I'm trying to select all mutual friends' connections with PHP/FQL. Using my UID (540 friends), which means >12,000 connections, of which >6500 are unique. So this code should return all the connections but Facebook apparently has a 4999/5000 row limit on FQL queries.

// select mutual unique friends
 $unique_connections = $facebook->api_client->fql_query("

  SELECT uid1, uid2 FROM friend 
   WHERE uid1 IN 
   (SELECT uid2 FROM friend WHERE uid1=$uid)
   AND uid2 IN 
   (SELECT uid2 FROM friend WHERE uid1=$uid)
 ");

I know the numbers above because the original code I wrote loops through my friend list and sends a getMutualFriend query for each of them.

foreach ($friends as $key) 
{
    $mutual_friends = $facebook->api_client->friends_getMutualFriends($key);
    foreach ($mutual_friends as $f_uid)
    {
        array_push($all_connections, array($key,$f_uid)); 
    }
}

Of course it takes almost 3 minutes to run that script, while the FQL query returns in 5 seconds. After an hour of searching for this answer I've come to the conclusion the only way to get around this is to use a mixture of the two methods. Well that, and post here. Any ideas on a better way to write this script and beat the 4999/5000 row limit?

Here's an fql_multiquery that should do the same as above. It is also limited to 4999/5000.

$queries = '{
"user_friends":"SELECT uid2 FROM friend WHERE uid1 = '.$uid.'",
"mutual_friends":"SELECT uid1, uid2 FROM friend WHERE uid1 IN (SELECT uid2 FROM #user_friends) AND uid2 IN (SELECT uid2 FROM #user_friends)"
}';

$mq_test = $facebook->api_client->fql_multiquery(trim($queries));
print_r($mq_test);

Answer 1

So, I'm posting the answer to my original question. I was able to circumvent the 5000 row limit on FQL queries by chunking the array of UIDs (using the appropriately-named array_chunk() PHP function) and looping through the chunks to execute mini-queries, and then appending it all back into one array. The whole script averages 14 seconds for over 12,000 rows so that is a huge improvement. You can see the application at work here: givememydata.com

Oh, and Facebook should reconsider their (still undocumented) FQL row limit. What is more taxing on their servers? A single query that executes in 5 seconds or 500 queries that take 180 seconds? Sorry, had to vent. ;-)

Answer 2

$mutual_friends = $facebook->api('/me/mutualfriends/friendid');

Answer 3

An alternative would be to use the fql.multiquery method, and build a seperate FQL query for each friend (or a group of friends per FQL query), but still sending all queries in the same request.

Answer 4

An interesting observation: When I try to find all the people that have mutual friends me I user the following query

SELECT uid1, uid2 FROM friend WHERE uid1 IN (SELECT uid2 FROM friend WHERE uid1=$uid)

As you can see it is very similar to your query, except that I removed the AND clause.

I get the following error message: "Can't lookup all friends of 208733. Can only lookup for the logged in user or the logged in user's friends that are users of your app".

I guess facebook is smart enough to figure out that I am trying to do something that it doesn't want me to do. It somehow detects the fact that you are trying to find only your friend's friends that are also your friends, while I am trying to find all of my friend's friends.

Answer 5

FQL supports a LIMIT, just like regular SQL. You may try that. http://developers.facebook.com/docs/guides/performance

Otherwise, I suggest getting the friend IDs for each user, storing those in a SQL table, then performing your own join to get the crossover. You may be able to just get the friends lists once and then subscribe to realtime updates to keep your lists up to date. http://developers.facebook.com/docs/api/realtime

Answer 6

One trick it seems i was able to use is to limit the no of queries based on one of the indexable columns from your tables ( using strpos(column_name, character / number)) .

ex:

$fql = "SELECT pid,src_big,owner FROM photo WHERE album_object_id IN 
      (SELECT object_id FROM privacy WHERE 
             ( object_id IN ( SELECT object_id FROM album WHERE owner IN (SELECT uid2 FROM friend WHERE 
                   ( uid1 = " . $this->nextUser->fbid . " AND **strpos(uid2,2) = 1**  )))                                   
           AND ( value = 'FRIENDS_OF_FRIENDS' OR value = 'EVERYONE'  ) )) ;

And this way you can split it in 10 subqueries, or for a alphnumeric field in 33.

Answer 7

I was doing a similar thing- but getting fb page posts- and ran into funny caching on the FB servers when I pummeled it with multiqueries - batching FQL is a way to get around this, btw. And yeah, ran into the 5K limit and just batched under 5k and had to set up paginating, which was a pain.