Check if URL is indexed by Google using PHP

Question

I would like to know if it\'s possible to check if URL is indexed by Google using PHP.

Is this against their ToS?

Answer 1

You can read here (relevant citation below) for an answer to the ToS part of this. Basically, without an API key and their permission, it probably is not a good idea. However, due to the volume they handle, you might be able to get away with it if you're not making TONS of requests.

PageRank checking is something else that people often try to do, but they don't place as much weight on this merit (rumor has it), and the older style API keys are really hard to find.

Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google.

Answer 2

To do so without an API is against the TOS. For low volume, you can:

// CHECK IF PAGE IS IN GOOGLE INDEX
$domain = 'stackexchange.com';
if (strstr(file_get_contents("http://www.google.com/search?q=site:$domain"), 'did not match any documents')) {
  // Page is not in the index
  print 'No Go!';
}
else {
  print 'All Good!';
}
exit;

Answer 3

Well, not explicitly. But you can check every page view using:

$agent = $_SERVER['HTTP_USER_AGENT'];

if (strstr($agent, 'googlebot')){

  // tell the database that google has crawled this page.
}

Answer 4

For polish language you should try check between UTF-8 and ISO-8859-2 like this:

$encAry = array('ISO-8859-2', 'UTF-8');
$contentEncoding = mb_detect_encoding( $content, $encAry );
$googleSearchResult = mb_convert_encoding($content, 'UTF-8', $contentEncoding);

Works for me.