Find Top 10 Most Frequent visited URl, data is stored across network

Question

Source: Google Interview Question

Given a large network of computers, each keeping log files of visited urls, find the top ten most visited URLs

Answer 1

In the worst case any algorithm, which does not require transmitting the whole frequency table, is going to fail. We can create a trivial case where the global top-10s are all at the bottom of every individual machines list.

If we assume that the frequency of URIs follow Zipf's law, we can come up with effecive solutions. One such solution follows.

Each machine sends top-K elements. K depends solely on the bandwidth available. One master machine aggregates the frequencies and finds the 10th maximum frequency value "V10" (note that this is a lower limit. Since the global top-10 may not be in top-K of every machine, the sum is incomplete).

In the next step every machine sends a list of URIs whose frequency is V10/M (where M is the number of machines). The union of all such is sent back to every machine. Each machines, in turn, sends back the frequency for this particular list. A master aggregates this list into top-10 list.