Crawling Google Search with PHP

Question

I trying to get my head around how to fetch Google search results with PHP or JavaScript. I know it has been possible before but now I can\'t find a way.

I am trying to

Answer 1

Odd. Because if I do a curl from the command like I get a 200 OK:

curl -I 'http://www.google.se/#hl=sv&q=dogs'
HTTP/1.1 200 OK
Date: Sun, 27 Jan 2013 20:45:02 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=b82cb66e9d996c48:FF=0:TM=1359319502:LM=1359319502:S=D-LW-_w8GlMfw-lX; expires=Tue, 27-Jan-2015 20:45:02 GMT; path=/; domain=.google.se
Set-Cookie: NID=67=XtW2l43TDBuOaOnhWkQ-AeRbpZOiA-UYEcs7BIgfGs41FkHlEegssgllBRmfhgQDwubG3JB0s5691OLHpNmLSNmJrKHKGZuwxCJYv1qnaBPtzitRECdLAIL0oQ0DSkrx; expires=Mon, 29-Jul-2013 20:45:02 GMT; path=/; domain=.google.se; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked

Also, maybe consider setting a urlencode for the passed URL so this line:

curl_setopt($ch, CURLOPT_URL, 'http://www.google.se/#hl=sv&q=dogs');

Changes to this:

curl_setopt($ch, CURLOPT_URL, 'http://www.google.se/' . urlencode('#hl=sv&q=dogs'));