Python - Easy way to scrape Google, download top N hits (entire .html documents) for given search?

Question

Is there an easy way to scrape Google and write the text (just the text) of the top N (say, 1000) .html (or whatever) documents for a given search?

As an example, imagin

Answer 1

Check out BeautifulSoup for scraping the content out of web pages. It is supposed to be very tolerant of broken web pages which will help because not all results are well formed. So you should be able to:

• Request http://www.google.ca/search?q=QUERY_HERE
• Extract and follow result links using BeautifulSoup (It appears as though class="r" for result links)
• Extract text from result pages using BeautifulSoup